Skip to main content

Posts

Beautiful Soup HTML parsing The following Python code fetches the specific windspeed web page and extracts the timestamp, average windspeed, direction, gust speed and writes out data to a date stamped file named say /home/user/wind_data/windspeed_date(2015-04-21-12).txt. Schedule a cron job to run this every day at midnight say. The windspeed file can be selected for a particular day and processed by graph.py.  #!/usr/bin/python import os import requests import time from bs4 import BeautifulSoup date_stamp = time.strftime('%Y-%m-%d-%H',(time.localtime(time.time()))) outfile = os.path.join(os.path.expanduser('~'), 'wind_data', "windspeed_%s.txt"%date_stamp) f = open(outfile,'w') list = [] r = requests.get("http://xxxxx.wwww.yyyyy") soup = BeautifulSoup(r.content) table = soup.find("table", {"id":"grid"}) for line in table.findAll('tr'): for l in line.findAll('td'):
Recent posts

Arcsight Logger 5.3 CentOS 5.9 Virtualisation

Get the VirtualBox CentOS 5 as a 7z file from: http://virtualboxes.org/images/centos/ Uncompress and save the Centos64.vbox and Centos64.vdi files. Open the Oracle VM VirtualBox manager and select the vbox file to install Make > 20GB disc or there will not be room for logger. Login as root/reverse Install Gnome Desktop as follows and start: yum groupinstall "X Window System" "GNOME Desktop Environment"  login root/reverse and startx Check version of CentOS and other prelim: cat /etc/redhat-release #CentOS release 5.9 (Final) uname -a # somewhere x86_64  create user logger open port 443 check logger bin file execute box and double click. run in terminal If not enough space to install logger: C:\Program Files\Oracle\VirtualBox\VBoxManage.exe modifyhd "D:\virtual machines\Centos\centos64.vdi" --resize 20000 shutdown centos VM attach gparted-live-0.16.2-1b-i486.iso to CD drive resize sda up to increased size  start Centos VM and

Splunk Cheat Sheet (Linux)

1. set root's password:  sudo su passwd root Enter new UNIX password: < new_root_password > Retype new UNIX password: < new_root_password > passwd: password updated successfully # su - 2. Remove any existing Splunk directories & create user etc: # rm -rf /opt/splunkforwarder # userdel -r splunk # this will remove as above if user splunk's home directory # groupadd siem # useradd -g siem -s /bin/bash -d /home/siem -m siem # vi ~/.profile # chage -I -1 -m -0 -M -99999 -E -1 siem If above fails because of multiple passwd fails: # pam_tally --reset check with #chage -l siem # uname -a # check OS version # dpkg -i splunk-4.3.1...........intel.deb # chown -R siem:siem /opt/splunk # su - siem : $SPLUNK_HOME/bin/splunk start --accept-license : $SPLUNK_HOME/bin/splunk edit user admin -password newpassword -role admin -auth admin:changeme 3. vi ~/.profile (as follows) (OR .bash_profile) # ~/.profile: executed by the command interpreter for log

Arcsight Syslog Connector Test - Scapy

This Python script uses Scapy to generate a UDP syslog frame, in Common Event Format (CEF), simulating a McAfee IPS output following detection of a Java heap buffer overflow attack. #! /usr/bin/env python # Note  need to be superuser or root access to run as Scapy sends # packets to the network card driver, an operation that users with ordinary # access are not permitted to do. from scapy.all import * a = IP(dst="192.168.1.2") u = UDP(dport=514) pay = "Aug 1 10:00:00 McAfee CEF:0|McAfee|Network Security Manager|Simulated|44800015|HTTP: Java heap buffer overflow detected|8|dvc=192.186.1.170" packet = a/u/pay packet.display() send(packet) The UDP packet is sent to the Arcsight CEF syslog connector at 192.168.1.2 for onward transmission to the Arcsight ESM. The Wireshark capture is as follows:

Splunk Universal Forwarder – Windows

To collect log information from a stand alone Windows (Vista in this case) workstation, download and install the Splunk Universal Forwarder splunk-4.3-115073-x86-release.msi Follow the install configuration and select the WindowsEventLog : Security and whatever performance monitoring is required. Enter the IP address of the Splunk indexer/search head and the host IP address as directed. If the default port (9997) was selected ensure that the indexer firewall is set to allow connections from the forwarder on this port. In addition to the standard windows event logs: Application log (tracks events that occur in a registered application) Security log (tracks security changes and possible breaches in security) System log (tracks system events) a couple of useful logs to forward are the windows update log and the windows firewall log. Start Notepad and run as administrator. Open C:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf and edit as follows:

QUISK SDR 40 metre RX

To test the previous GNU radio apps and evaluate the linux SDR QUISK software, the following 7MHz SDR was quickly assembled from available parts.  The heart of the hardware is the modulator section of a Marconi QPSK modem using Watkins Johnson M6E mixers with a 90 deg phase shift circuit modified for the xtal frequency of 7035kHz. Using the Softrock .quisk_conf.py file for QUISK with: fixed_vfo_freq = 7035000        sample_rate = 96000 The following daytime spectrograph was obtained on 40 metres. The SDR hardware (90% not used, including the 741 or TL081 op amps as they were noisy + insufficient gain-BW). The PC was a Toshiba Satellite L650 laptop with Ubuntu 11.10 OS.

GNU Radio Waterfall and CW Filter

The following GNU radio application adds a waterfall spectrogram to the previous CW filter program. The plot show 4 CW signals in the audio band (lower sideband) at 7023 kHz. The 700Hz signal is filtered and output to the laptop headphones by the CW bandpass filter. The frequency display is shown after the script which is as follows: #!/usr/bin/env python from gnuradio import gr from gnuradio import audio from lpf_bpf_class import Bandpass from gnuradio.qtgui import qtgui from PyQt4 import QtGui import sys, sip     class cw_filter(gr.top_block):     def __init__(self):         gr.top_block.__init__(self)           sample_rate = 44100         out_rate = 8000         kaiser = Bandpass()         cw_flr = gr.fir_filter_fff(1, kaiser.bpftaps)         decimate = int(sample_rate/out_rate)         Bandpass.cutoff1 = 3000                pre_decim = Bandpass()         dec_flr = gr.fir_filter_fff(1, pre_decim.lpftaps)         dec = gr.keep_one_in_n(gr.sizeof_float, decima