Category > Guides

Building a Python Coding Environment in Windows

» 07 February 2011 » In Guides, Life, Programming, Python » 8 Comments

To be honest, my computer files are not organized. Yes, and you know what? Most of my scripts (Perl and Python) sits on one directory: C:\Perl64\scripts. And inside that directory, contains mixed files of Perl scripts, Python Scripts, output folders, text files, dictionaries, CSV files, etc. It’s sad that I can’t find the time motivation to organize my scripts.

Well, I came up with this pretty neat trick. I used it in Java, but for this post, I’ll be applying it in Python. I will place all of my Python scripts in my personal drive R:\ and under the folder R:\Python. Well, that sounds too simple doesn’t it? Let me expand the idea.

I’m on Windows, and yes I have ArchLinux on dual-boot, but I’ll apply this to Windows for now. One great reason why I love coding in Windows, is because I love Notepad++. Sadly, I’m not gifted with the ability to write code on Vim, or E-macs (I haven’t tried). And oh, my current Notepad++ style is Obsidian, and it’s just so comfortable to use.

Anyway, as I’ve mentioned, I’m on Windows. But for example I’m browsing on my favorite browser (Chrome), and I just had an idea for a neat, cool script. So I’ll fire up notepad++, change the language to Python, then fire up cmd, cd to my Python directory, and start to code (and debug). But wait, that sounds too long, well for me. It’s kind of boring that I have to do that repetitive process over and over again, every single day. Plus, the files is pretty much unorganized. What to do now? Well of course, hack it!

Now, let’s get back to the situation. Say I’m browsing the interwebs (with my favorite browser Chrome) and I got an idea for the script. I would normally open notepad++ first, but this time, I think it would be better to access the python directory first with cmd. The quickest way to access cmd, wherever you are, is by the Run Dialog (That’s Win+R for the noobs). Just type in cmd and you’re done. But I have to cd to my python directory, and re-enter the drive if it’s not on C:\, that’s 2 commands already! Now, we can fix this, and we need a script of course. For this one we can use a batch script. Now now, many people say that this is old school, but let’s face it, we’re on Windows.

@echo off
echo.
echo Time for some pure python awesomeness!
echo.
cmd /k "cd R:\Python && R:"

Save it with any filename you wish. For me, I saved it as pyc.bat in the Python bin folder (make the directory is included in your PATH directory), so I can just type pyc on the Run Dialog box.

We’re done on the cmd part, now for notepad++. For this, the script we’ll be using is a python script. And to start, create a new script located on the Python script folder.

#!/usr/bin/python

'''
	This script is used to create new (organized)
	Python scripts
'''

import os
import sys

def main(arg):
	fdir = arg
	arg += '.py'
	
	#Create directory for the script.
	os.mkdir(fdir)
	
	#Create the file inside the directory
	fo = open('%s\\%s' % (fdir, arg), 'w')
	#Change or add your own lines if you want.
	fo.write('#!/usr/bin/python')
	fo.close()
	
	#Open the script in Notepad++
	#If your notepad++ installation directory is different,
	#change it.
	notepad = '"C:\\Program Files (x86)\\Notepad++\\notepad++.exe" '
	os.system('%s%s\\%s' % (notepad, fdir, arg))
	os.system('cmd /k "cd "%s"' % fdir)
	
if __name__ == '__main__':
	main(sys.argv[1])

To use it, simple do something like:

R:\Python>new coolscript

And it will create a directory based on the parameter you passed, and a script inside it. And opens it in Notepad++, also it will make the cmd cd to the newly created directory. The only issue here is new.py is visible to the naked eye in the Python scripts folder. To hide the file, just do:

attrib +S +H +R new.py

And it will remain hidden (unless you chose to show system files in folder options). S means system, H means hidden, and R means read only.

TRIVIA: It’s how most malware hide themselves in Windows.

Now, we have a good, organized python coding environment in Windows. If you have any comments or suggestions, just drop a comment. Cheers!

Continue reading...

Tags: , , , ,

Using GAE Cron to get SMS Alerts from Gmail

» 06 February 2011 » In Guides, Internet, Programming, Python » 13 Comments

Who doesn’t love the free email service Gmail? I am currently using Google Apps Mail for this domain, and I got it installed in my computer, so I just have to switch windows to check messages. Most of the emails I am receiving is crucial, especially from Scriptlance. And when I am offline (either I am sleeping or I am not home), I don’t know if an email arrives, or if I should be online to respond to that email. So the solution for this? SMS alert.

I’m pretty sure there are lots of services that offer email alerts to your phone, but chances are, they are not free, and we are not sure if the login information is kept or used in something we do not like.

For this to work, there are some requirements:

  • SMS Gateway
  • Google App Engine Account
  • Google App Engine SDK (and Python 2.5 of course)
  • Your Gmail or Google Apps Mail credentials

The SMS gateway is not free in most cases, but you can use if a friend has one, just make sure it works with your carrier.

The first thing you need to do is create a GAE Application. Name it whatever you like, and on the app.yaml, add these lines:

- url: /check/mail
  script: check.py
  login: admin

Of course you can changes the url and script, but I just like to use that in this guide. Next is to create cron.yaml, and put the following lines:

cron:
- description: Check Mail
  url: /check/mail
  schedule: every 1 minutes

You can change the schedule on how frequent you like your mail to be checked. For me, I chose to check every one minute (There’s a grammatical error on the yaml file, but ignore it, it’s correct according to the syntax). That means, every minute, the script will check for unread messages and sends me a SMS message if there is one.

Next is to create a script called check.py:

#!/usr/bin/env python
'''
	Copyright (c) Ruel Pagayon <http://ruel.me>
'''

from xml.dom.minidom import parseString
from google.appengine.ext import db
import urllib2
import urllib
import base64
import sys

'''
	Change the values below.
'''
gFeed = 'https://mail.google.com/mail/feed/atom'
#gFeed = 'https://mail.google.com/a/domian.tld/feed/atom'	#For Google Apps for Mail users.

gEmail = 'user@domain.tld'
gPass = 'pass'

class Uid(db.Model):
	cont = db.StringProperty()

def sendSMS(message):
	'''
		Insert appropriate code for your SMS gateway here.
		Obviously the one below doesn't work.
		It's there to give you an insight on how it will work.
	'''
	urllib2.urlopen('http://somegateway.com:1578/?user=smsuser&password=smspass&PhoneNumber=18005669874&Text=%s' % urllib.quote(message))
	
def getNewMail(feed):
	'''
		Check for new unread mails. Use basic auth.
	'''
	request = urllib2.Request(feed)
	base64string = base64.encodestring('&s:%s' % (gEmail, gPass)).replace('\n', '')
	request.add_header("Authorization", "Basic %s" % base64string)
	'''
		Parse XML using DOM
	'''
	xdata = urllib2.urlopen(request).read()
	dom = parseString(xdata)
	count = dom.getElementsByTagName('fullcount')[0].firstChild.data
	
	'''
		Retrieve last message ID from datastore
	'''
	ouid = Uid.all()
	
	if ouid.count() != 0:
		lastId = ouid[0].cont
		
	if int(count) > 0:
		'''
			Check if there are feed entries.
			Get the ID of the latest message.
		'''
		eid = dom.getElementsByTagName('entry')[0].getElementsByTagName('id')[0].firstChild.data
		'''
			Generate report.
		'''
		message = "You got " + count + " new email(s)\n\n"
		for entry in dom.getElementsByTagName('entry'):
			'''
				Loop through each email entry
			'''
			message += "Sender: %s (%s)\n" % (entry.getElementsByTagName('name')[0].firstChild.data, entry.getElementsByTagName('email')[0].firstChild.data)
			message += "Subject: %s\n\n" % entry.getElementsByTagName('title')[0].firstChild.data
			
		if eid != lastId:
			'''
				If the last ID from the data store, 
				is different from the latest message ID,
				update the data store and
				send the SMS
			'''
			if ouid.count() != 0:
				nuid = Uid.get(ouid[0].key())
			else:
				nuid = Uid()
			nuid.cont = eid
			nuid.put()
			sendSMS(message)
		'''
			You got 1 new email(s)
			
			Sender: Someone (someone@domain.tld)
			Subject: Example Message
		'''

def main():
	getNewMail(gFeed)

if __name__ == '__main__':
	main()

Change the values that suites your needs. And the docstrings on the script pretty much explains everything, so I don’t have to repeat it here.

Now, we all know that a free GAE app has quota, and on the script, we will be using the Data Store API and the UrlFetch API, let’s do some math. The quota will reset every 24 hours. So, if you decided to check the mail every 1 minute, we got 1440 calls. And it’s not even a quarter of the limit.

Now obviously we can use the script on our webservers, the only thing that we need to change is the data store calls (GAE doesn’t allow writing to disk, so Datastore is used). But the problem here is the credentials are in plain text, and there’s a huge risk for this information to be seen by other parties. So why not just use GAE? Where only you and Google can have access. :)

Continue reading...

Tags: , , , , , ,

Using Google Fonts Locally

» 03 February 2011 » In Guides, Web Design » 15 Comments

Google Fonts, or more popularly known as Google Web Fonts, provides a great collection of open source licensed (can be used commercially and non-commercially) true-type fonts. My favorite is Droid Sans, the font I’m currently using now sitewide.

Because I love Droid Sans, I want to use it everywhere. Even in my school projects. I just finished a project (that is due tomorrow) that demonstrates HTML frames. And I wan’t (of course) to use Droid Sans as the main font for the web page. And if I used the font as I regularly do:

<link href='http://fonts.googleapis.com/css?family=Droid+Sans:regular&amp;subset=latin' rel='stylesheet'> 

The font won’t be viewable offline. So the solution for this, is to download the font and use the CSS Google recommends. Take a look here.

@media screen {
@font-face {
  font-family: 'Droid Sans';
  font-style: normal;
  font-weight: normal;
  src: local('Droid Sans'), local('DroidSans'), url('http://themes.googleusercontent.com/font?kit=s-BiyweUPV0v-yRb-cjciC3USBnSvpkopQaUR-2r7iU') format('truetype');
}
}

But how do we download the font? That’s easy look at the url('...'):

src: local('Droid Sans'), local('DroidSans'), url('http://themes.googleusercontent.com/font?kit=s-BiyweUPV0v-yRb-cjciC3USBnSvpkopQaUR-2r7iU') format('truetype');

Just enter that URL to your browser, and viola!

Now that we got our font, we use the CSS, and we modify it a little bit:

@font-face {
  font-family: 'Droid Sans';
  font-style: normal;
  font-weight: normal;
  src: local('Droid Sans'), local('DroidSans'), url('path/to/DroidSans.ttf') format('truetype');
}

So this works for the latest stable versions of Chrome, Firefox, Opera, Safari, and.. Wait.. It’s not working in IE8. This is because IE lte 8 only accepts EOT type fonts. So how do we fix this? I’m not an IE user, but a big part of the Internet population uses IE, so we do not have a choice.

Good news is, you can convert TTF fonts to EOT. After that, save the file on the same directory where our TTF font is, and add it in the CSS.

@font-face {
  font-family: 'Droid Sans';
  font-style: normal;
  font-weight: normal;
  src: url('path/to/DroidSans.eot');
  src: local('Droid Sans'), local('DroidSans'), url('path/to/DroidSans.ttf') format('truetype');
}

Now, you have to remember that you have to place this on the top of your CSS document. Then you can just use this as a regular font in CSS:

body {
	font-family: 'Droid Sans', Arial, sans;
	color: #666;
}

Make sure, that the path to the font is always relevant to the CSS file. And that’s all for now. :)

Continue reading...

Tags: , , , , ,

SSH Tunneling with Putty

» 27 January 2011 » In Guides, Internet, Python, Security » 2 Comments

Privacy and anonymity, these two does not exist when you’re connected to the internet. Yes, almost all of the internet users think that their actions/history is not recorded. The sad truth is, every single data that passes through your ISP’s cables is recorded. Now, there are still people thinking they can be anonymous, if they’re behind proxies etc. I myself is one of those people, but that was 2 years ago.

I used to be behind SOCKS proxies, and I thought I can do whatever I want. But since they are highly anonymous, SOCKS proxies are uber slow. And then, I was introduced to SSH Tunneling.

What’s the use?

Yes, you might notice, I said privacy and anonymity doesn’t exist in the internet. But now, we are more concerned about the other end of the connection. Since the main purpose of this, is to hide our true geographic information from the other end.

Another reason, is bypassing restrictions on the current network. Using proxies or tunnels, you can access restricted websites and protocols. One example is on school. In our university, there are loads of restricted websites, mostly social networking, and even proxy providers. Though most restricted websites can be accessed with SSL, not all provides encryption, so a tunnel is much more preferable.

Advantages of SSH Tunnels

To summarize, SSH Tunnels are much more secure and faster. And unlike proxies, you have access to the actual server, so you have control over the logs, etc.

How is it done?

Before going to the actual procedures, you must first have the following requirements.

Yes, you need a SSH Account on a remote web server. Luckily for me, my good friend Andy allowed me to have a space on his VPS, and I have my own account there.

So the first thing you need to do is to fire up putty.

Putty Session Page

Putty Session Page

Enter the server’s host address and the assigned SSH port (22 is the default).

Putty SSH Tunnel

Putty SSH Tunnel

Then, on the left side, go to Connection -> SSH -> Tunnels. Add a new forwarded port, I suggest those above 1024, and Click the Dynamic radio button. Then click add. Open the session, login, and you’re done.

Application

Now that we have our set up, we can use this just like a regular SOCKS proxy.

Host: 127.0.0.1
Port: 8080 (or whichever port you forwarded)

You can verify your connection by visiting websites that tells your IP. An example is: whatsmyip.

If you have problems or clarifications about this topic, feel free to comment.

Continue reading...

Tags: , , , , ,

DHCP on Cisco Packet Tracer

» 10 November 2010 » In Guides » 2 Comments

DHCP means Dynamic Host Configuration Protocol. and here’s a wiki entry about it:

The Dynamic Host Configuration Protocol (DHCP) is an auto configuration protocol used on IP networks. Computers that are connected to IP networks must be configured before they can communicate with other computers on the network. DHCP allows a computer to be configured automatically, eliminating the need for intervention by a network administrator. It also provides a central database for keeping track of computers that have been connected to the network. This prevents two computers from accidentally being configured with the same IP address.

I created a short video on how to configure a router to use DHCP on Cisco Packet Tracer. Its actually pretty simple, all you need is a couple of commands, here’s the video:

The commands in the video (if you didn’t catch it right) are:

  • ip dhcp exc 192.168.1.1 192.168.1.99
  • ip dhcp pool P1
  • network 192.168.1.0 255.255.255.0
  • default-router 192.168.1.1

I suggest you try it yourself. Good luck!

Continue reading...

Tags: , , , ,

Web Page Scraping with Python

» 06 November 2010 » In Guides, Python » 6 Comments

Whenever we come across a web page, we read its content, and even save it for later reading and analysis. Some web pages offer a lot of good information that maybe useful later. And sometimes those information are not easy to copy. It will require a lot of patience especially when the information you want to extract from a web page is on a bad format. Luckily, there’s what we call scraping, and of course to make things faster, programming will kick in.

Say for example, we need to harvest some email addresses from a web page. Lets use this page as an example. It’s a webpage with several email addresses, and our goal is to copy them all. Of course it will be a bit easy if we just use our mouse and copy-paste the email addresses. But the thing is, what if there’s another page to be scraped, with thousands of unique email addresses. That would be a pain, wouldn’t it?

The way we will extract data, will be regular expressions. It’s very popular and over used, but I believe it would be the best tool to accomplish this. And for the language, we will use Python. To be honest, I haven’t learned Python very well. I’m more like a Perl guy, but yeah, it’s time to learn something new. My version of Python is 2.5, you can use the latest 2.7 but not 3.x. Actually you can too, but I don’t recommend it. Most Linux distributions have Python installed, just like Perl. And for windows, you can download ActivePython.

The first thing we need, is to find/write a regular expression for the harvest. Now you can just search Google and find the one you need, here’s an example:

([A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4})

Let me explain this regular expression first, before we continue. The [A-Z0-9._%+-] part means match any of the following: A-Z, 0-9, a dot (.), underscore(_), percent sign (%), plus sign (+) and minus sign (-). The following + means match one or more of the set of characters inside the brackets []. So basically, [A-Z0-9._%+-]+ matches the local part of the email address. Next is the @ character. It’s pretty literal, because it means, match the @ character once, after the local part. The [A-Z0-9.-]+\.[A-Z]{2,4} matches the domain.

Here’s the code in Python, this prints out all the email addresses found on our example page.

#!/usr/bin/python

import urllib2
import re

request = urllib2.Request('http://www.daniweb.com/forums/thread218079.html')
request.add_header('UserAgent', 'Ruel.ME Sample Scraper')
response = urllib2.urlopen(request)
for line in response.read().split('\n'):
	match = re.search(r'([A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4})', line, re.I)
	if match:
		print match.group(1)

I will explain the code line by line:

  • import urllib2 import the urlib2 module for our connection
  • import re import the regular expression module
  • request = urllib2.Request('http://www.daniweb.com/forums/thread218079.html') make the request of the page
  • request.add_header('UserAgent', 'Ruel.ME Sample Scraper') add a UserAgent
  • response = urllib2.urlopen(request) performs the request
  • for line in response.read().split('\n'):(request) read the html source line by line
  • match = re.search(r'([A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4})', line, re.I) perform the search with the regular expression
  • if match: check if there are matches
  • print match.group(1) display the match

It was actually pretty easy. So the output of the script is the email addresses from that page. Web page scraping is not a hard task. It’s very easy and exciting. You can use other languages like Perl, PHP, etc. With this methods, data extraction is easier, and faster.

Continue reading...

Tags: , , , , , ,