Open Source Tools for Chapter 1 ***begin code*** c:\>montage -help ***end code*** Open Source Tools for Chapter 1 ***begin code*** c:\cygwin64\bin>wc c:\ftp\simplify.txt output: 10718 123383 849223 c:\ftp\simplify.txt ***end code*** Open Source Tools for Chapter 1 ***begin code*** copy diener.txt f: copy simplify.txt f: copy re-ana.txt f: copy phenocop.txt f: copy mystery.txt f: copy disaster.txt f: copy factnote.txt f: copy perlbig.txt f: copy create.txt f: copy exploreo.txt f: ***end code*** Open Source Tools for Chapter 1 ***begin code*** c:\ftp>fback.bat ***end code*** Open Source Tools for Chapter 1 ***begin code*** convert c:\ftp\eqn.jpg -contrast -contrast c:\ftp\results.png cd c:\cygwin64\home\E-Rock\ imdisplay c:\ftp\results.png cd c:\ftp ***end code*** Open Source Tools for Chapter 1 ***begin code*** c:\ftp>showpic.bat c:\ftp\eqn.jpg ***end code*** Open Source Tools for Chapter 1 ***begin code*** pause...Displays image file from input parameter. Must include file path... cd c:\cygwin64\home\E-Rock\ imdisplay %1 cd c:\ftp\ ***end code*** Open Source Tools for Chapter 1 ***begin code*** cd C:\ftp\back\aurora if not "%1" == "" goto start a.exe c:\ftp\perlbig.txt goto end :start a.exe c:\ftp\%1 :end cd c:\ftp exit ***end code*** Open Source Tools for Chapter 1 ***begin code*** $ chmod +x mybashfile ***end code*** Open Source Tools for Chapter 1 ***begin code*** $ mybashfile infile.gif out.gif -s 5 -d 5 -c 5 -g 1 -p 2 -b white ***end code*** Open Source Tools for Chapter 1 ***begin code*** $ mybashfile -h ***end code*** Open Source Tools for Chapter 1 ***begin code*** c:\>python Python 3.3.2 (v3.3.2:d047928ae3f6, May 16 2013, 00:06:53) [MSC v.1600 64 bit (AM D64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> print(5 + 5) 10 >>> ***end code*** Open Source Tools for Chapter 1 ***begin code*** c:\>irb irb(main):001:0> puts (5+5) 10 => nil ***end code*** Open Source Tools for Chapter 1 ***begin code*** c:\>perl -e "print 5+5"; 10 ***end code*** Open Source Tools for Chapter 1 ***begin code*** c:\ftp>perl -d -e 1 ***end code*** Open Source Tools for Chapter 1 ***begin code*** c:\ftp>perl -d -e 1 Loading DB routines from perl5db.pl version 1.33 Editor support available. Enter h or `h h' for help, or `perldoc perldebug' for more help. main::(-e:1): 1 DB<1> print(gmtime()); 3937203411501220 DB<2> print(1+6); 7 DB<3> print(4/0); Illegal division by zero at (eval 11)[C:/Perl64/lib/perl5db.pl:640] line 2. ***end code*** Open Source Tools for Chapter 1 ***begin code*** $ pip install rdflib (on Linux shell) c:\>pip install rdflib (on Windows) ***end code*** Open Source Tools for Chapter 1 ***begin code*** c:\>cpan ***end code*** Open Source Tools for Chapter 1 ***begin code*** cpan> install Chart::Gnuplot ***end code*** Open Source Tools for Chapter 1 ***begin code*** c:\>gem install sqlite3-ruby -v=1.2.3 ***end code*** Open Source Tools for Chapter 1 ***begin code*** #!/usr/bin/perl system("dir"); exit; ***end code*** Open Source Tools for Chapter 1 ***begin code*** #!/usr/local/bin/python import os os.system("dir") exit ***end code*** Open Source Tools for Chapter 1 ***begin code*** #!/usr/bin/ruby system("Identify -verbose c:/ftp/3320_out.jpg >myimage.txt") exit ***end code*** Open Source Tools for Chapter 1 ***begin code*** #!/usr/local/bin/perl $file = "try2.gif"; system("convert -size 350x40 xc:lightgray -font Arial -pointsize 32 -fill black -gravity north -annotate +0+0 \"Hello, World\" $file"); exit; ***end code*** Open Source Tools for Chapter 1 ***begin code*** #!/usr/local/bin/python import os, re, string filelist = os.listdir(".") for file in filelist: if ".jpg" in file: img_in = file img_out = "f:/" + file command = "convert " + img_in + " -set colorspace Gray -separate -average " + img_out os.system(command) exit ***end code*** Open Source Tools for Chapter 1 ***begin code*** hi.py #!/usr/local/bin/python print("Hi, I'm a Python script") exit hi.pl #!/usr/local/bin/perl print "Hi, I'm a Perl script\n"; exit; hi.rb #!/usr/local/bin/ruby puts "Hi, I'm a Ruby script" exit ***end code*** Open Source Tools for Chapter 1 ***begin code*** #!/usr/local/bin/ruby system("hi.py") system("hi.pl") system("hi.rb") exit ***end code*** Open Source Tools for Chapter 1 ***begin code*** c:\ftp>call_everyone.rb Hi, I'm a Python script Hi, I'm a Perl script Hi, I'm a Ruby script ***end code*** Section 2.2. ***begin code*** 1. X_HIST.PL 2. xlxs_spreadsheet.pdf 3. XMLVOCAB.PL 4. XOXOHWRD.ZBK 5. xy.TXT 6. XY_RAND.PL 7. xyrite.TXT ***end code*** Section 2.2. ***begin code*** 1. xlxs_spreadsheet.pdf 2. XMLVOCAB.PL 3. XOXOHWRD.ZBK 4. xy.TXT 5. xyrite.TXT 6. XY_RAND.PL 7. X_HIST.PL ***end code*** Section 2.2. ***begin code*** XMLVOCAB.PL XOXOHWRD.ZBK XY_RAND.PL X_HIST.PL xlxs_spreadsheet.pdf xy.TXT xyrite.TXT ***end code*** Section 2.2. ***begin code*** #!/usr/local/bin/perl @word_array = ("MacIntire", "Macadam", "wilson", "tilson", "Wilson", "I cannot go", "I can not go", "I can also go", "I candle maker", "O'Brien", "OBrien", "O'Brien's", "O'Briens", "OBrien's", "Oar", "O'Brienesque"); @word_array = sort (@word_array); print join("\n", @word_array); exit; ***end code*** Section 2.2. ***begin code*** C:\ftp\pl>perl sortlet.pl I can also go I can not go I candle maker I cannot go MacIntire Macadam O'Brien O'Brien's O'Brienesque O'Briens OBrien OBrien's Oar Wilson tilson wilson ***end code*** Section 2.2. ***begin code*** Ä (ASCII 142) ä (ASCII 132) A (ASCII 65) a (ASCII 97) á (ASCII 160) å (ASCII 134) ***end code*** Section 2.2. ***begin code***


à à
á á
â â
ã ã
ä ä
å å
æ æ
ç ç
è è
é é
ê ê
ë ë
ì ì
í í
î î
ï ï

***end code*** Section 2.2. ***begin code*** !"#$%&'()*+,-./0123456789:;<=> ?@ABCDEFGHIJKLMNOPQRSTUVWXYZ [\]^_`abcdefghijklmnopqrstuvwxyz{|}~ ***end code*** Section 2.2. ***begin code*** #!/usr/local/bin/perl $var = "µ½¼ßüéâäàèïÇÉúêæ That's all folks!"; $var =~ tr/\012\015\040-\176/ /c; print $var; exit; ***end code*** Section 2.2. ***begin code*** c:\ftp>unprintable.pl That's all folks! ***end code*** Section 2.2. ***begin code*** #!/usr/local/bin/python # -*- coding: iso-8859-15 -*- import string in_string = "prinüéêçâäàtable" out_string = filter(lambda x: x in string.printable, in_string) print out_string exit ***end code*** Section 2.2. ***begin code*** c:\ftp>printable.py printable ***end code*** Section 2.2. ***begin code*** #!/usr/local/bin/ruby puts "something here".tr('^a-z0-9\n!@#$%^&*()_-+=~`{[}]|":;?/.,> <\\\'','') exit ***end code*** Section 2.2. ***begin code*** #!/usr/local/bin/perl $var = "µ½¼ßüéâäàèïÇÉúêæ"; $var =~ s/×/ x /g; $var =~ s/–/\-/g; $var =~ s/—/\-/g; $var =~ s/"/\"/g; $var =~ s/"/\"/g; $var =~ s/’/\'/g; $var =~ s/‘/\'/g; $var =~ s/µ/ microns /g; $var =~ s/½/ one-half /g; $var =~ s/¼/ one-quarter /g; $var =~ s/ß/ beta /g; $var =~ s/æ/ae/g; $var =~ tr/üéâäàèïÇÉúê/ueaaaeiCEue/; $var =~ tr/\012\015\040-\176/ /c; print "The input, \"µ½¼ßüéâäàèïÇÉúêæ\" has been modified to \"$var\""; exit; ***end code*** Section 2.3. ***begin code*** By all means, pull the emergency cord if you want us all to die! ***end code*** Section 2.3. ***begin code*** #!/usr/local/bin/perl $all_text = "And, has thou slain the Jabberwock? Come to my arms, my beamish boy! O frabjous day! Callooh! Callay! He chortled in his joy. Lewis Carroll, excerpted from Jabberwocky"; $all_text =~ s/\n/ /g; $all_text =~ s/([^A-Z]+[\.\!\?][ \n]{1,3})([A-Z])/$1\n$2/g; print $all_text; exit; ***end code*** Section 2.3. ***begin code*** c:\ftp\pl>sentence.pl And, has thou slain the Jabberwock? Come to my arms, my beamish boy! O frabjous day! Callooh! Callay! He chortled in his joy. Lewis Carroll, excerpted from Jabberwocky ***end code*** Section 2.3. ***begin code*** #!/usr/local/bin/python import re all_text =\ "And, has thou slain the Jabberwock? Come \ to my arms, my beamish boy! O frabjous \ day! Callooh! Callay! He chortled in his \ joy. Lewis Carroll, excerpted from \ Jabberwocky"; sentence_list = re.split(r'[\.\!\?] +(?=[A-Z])', all_text) print '\n'.join(sentence_list) exit ***end code*** Section 2.3. ***begin code*** #!/usr/local/bin/ruby all_text =\ "And, has thou slain the Jabberwock? Come \ to my arms, my beamish boy! O frabjous \ day! Callooh! Callay! He chortled in his \ joy. Lewis Carroll, excerpted from \ Jabberwocky"; all_text.split(/[\.\!\?] +(?=[A-Z])/).each {|phrase| puts phrase} exit ***end code*** Section 2.5. ***begin code*** This book, Data Simplification, is way cool. ***end code*** Section 2.5. ***begin code*** June 16, 1904 Thoory Pyramus and Thisbe ***end code*** Open Source Tools for Chapter 2 ***begin code*** g Match globally, (find all occurrences). i Do case-insensitive pattern matching. m Treat string as multiple lines. o Compile pattern only once. s Treat string as single line. x Use extended regular expressions. ^ Match the beginning of the line. . Match any character (except newline). $ Match the end of the line (or before newline at the end). | Alternation. () Grouping. [] Character class. * Match 0 or more times. + Match 1 or more times. ? Match 1 or 0 times. {n} Match exactly n times. {n,} Match at least n times. {n,m} Match at least n but not more than m times. \n newline(LF, NL). \W Match a non-word character. \s Match a whitespace character. \S Match a non-whitespace character. \d Match a digit character. \D Match a non-digit character. ***end code*** Open Source Tools for Chapter 2 ***begin code*** $string =~ s///options; ***end code*** Open Source Tools for Chapter 2 ***begin code*** $string =~ s/[\n]+/ /g; $string =~ s/([^A-Z]+\.[ ]{1,2})([A-Z])/$1\n$2/g; ***end code*** Open Source Tools for Chapter 2 ***begin code*** #!/usr/local/bin/perl undef($/); open (TEXT, "english_lit.txt"); open (OUT, ">english_lit_names.txt"); $line = ; close TEXT; while ($line =~ /\b[A-Z][a-z]+[ \n]{1}[A-Z][a-z]+\b/g) { $name = $&; $name =~ s/\n/ /; next if ($name =~ /^The/); next if ($name =~ /^In/); next if ($name =~ /^Of/); next if ($name =~ /^In/); next if ($name =~ /^In/); next if ($name !~ /[A-Z][a-z]+/); $namelist{$name} = ""; } while ($line =~ /\b[A-Z][a-z]+[\, \n]+[A-Z][\,\. \n]*[A-Z]?[\,\. \n]*[A-Z]?[\,\. \n]*\b/g) { $name = $&; next if ($name =~ /^The/); next if ($name =~ /^In/); $name =~ s/[\,\.\n]//g; next if ($name =~ / {3,}/); $name =~ s/ +$//; next if ($name !~ /[A-Z][a-z]+/); $namelist{$name} = ""; } print OUT join("\n", sort(keys(%namelist))); close OUT; system 'notepad english_lit_names.txt'; exit; ***end code*** Open Source Tools for Chapter 2 ***begin code*** Abbess Hilda About Project Abraham Cowley Abraham Cowper Abt Vogler Academy Classics Adam Bede Adam Smith Adelaide Procter Adelaide Witham Ado About After Alfred Again Beowulf Albion Series Aldine Edition Aldine Poets Aldine Series Alexander Pope Alexander Selkirk Alfred Lord Alfred Tennyson Algernon Charles Alice Brown All Delight Alloway Kirk Although Bacon American Book American Indians American Revolution American Taxation Amerigo Vespucci Among Browning Among Coleridge Among My Among Ruskin Amos Barton ***end code*** Open Source Tools for Chapter 2 ***begin code*** %% a percent sign %c a character with the given number %s a string %d a signed integer, in decimal %u an unsigned integer, in decimal %o an unsigned integer, in octal %x an unsigned integer, in hexadecimal %e a floating-point number, in scientific notation %f a floating-point number, in fixed decimal notation %g a floating-point number, in %e or %f notation ***end code*** Open Source Tools for Chapter 2 ***begin code*** DB<1> printf("%-10.10s %0.1u %7.6u %4.3u\n", "hello", 3, 28, 15, ); ***end code*** Open Source Tools for Chapter 2 ***begin code*** hello 3 000028 015 ***end code*** Open Source Tools for Chapter 2 ***begin code*** >>> "%-20.20s %8.06d" % ("hello", 35) ***end code*** Open Source Tools for Chapter 2 ***begin code*** 'hello 000035' ***end code*** Open Source Tools for Chapter 2 ***begin code*** #!/usr/local/bin/ruby freq = Hash.new(0) my_string = "Peter Piper picked a peck of pickled peppers. A peck of pickled peppers Peter Piper picked. If Peter Piper picked a peck of pickled peppers, Where is the peck of pickled peppers that Peter Piper picked?" my_string.downcase.scan(/\w+/){|word| freq[word] = freq[word]+1} freq.sort_by {|k, v| v}.reverse.each {|k,v| printf "%-10.10s %0.2u\n", k, v} exit ***end code*** Open Source Tools for Chapter 2 ***begin code*** c:\ftp>printf_ruby.rb peter 04 piper 04 picked 04 peck 04 of 04 peppers 04 pickled 04 a 03 if 01 where 01 is 01 the 01 that 01 ***end code*** Open Source Tools for Chapter 2 ***begin code*** #!/usr/local/bin/ruby require 'base64' data = "Time flies like an arrow. Fruit flies like a banana." base64_transformed_data = Base64.encode64(data) puts base64_transformed_data base64_reverse = Base64.decode64(base64_transformed_data) puts base64_reverse exit ***end code*** Open Source Tools for Chapter 2 ***begin code*** c:\ftp>base64_ruby.rb VGltZSBmbGllcyBsaWtlIGFuIGFycm93LiAgRnJ1aXQgZmxpZXMgbGlrZSBh IGJhbmFuYS4= Time flies like an arrow. Fruit flies like a banana. ***end code*** Open Source Tools for Chapter 2 ***begin code*** #!/usr/local/bin/perl use MIME::Base64; open (TEXT,"gettysbu.txt"); binmode TEXT; $/ = undef; $string = ; close TEXT; $encoded = encode_base64($string); print $encoded; $decoded = decode_base64($encoded); print "\n\n$decoded"; exit; ***end code*** Open Source Tools for Chapter 2 ***begin code*** c:\ftp>base64_perl.pl Rm91ciBzY29yZSBhbmQgc2V2ZW4geWVhcnMgYWdvIG91ciBmYXRoZXJzIGJyb3VnaHQgZm9ydGgg b24gdGhpcw0KY29udGluZW50IGEgbmV3IG5hdGlvbiwgY29uY2VpdmVkIGluIGxpYmVydHkgYW5k ***end code*** Open Source Tools for Chapter 2 ***begin code*** #!/usr/local/bin/python import base64 sample_file = open ("gettysbu.txt", "rb") string = sample_file.read() sample_file.close() print base64.encodestring(string) print base64.decodestring(base64.encodestring(string)) exit ***end code*** Open Source Tools for Chapter 2 ***begin code*** #!/usr/local/bin/perl opendir(XPDF_SUBDIR, ".") || die ("Unable to open directory"); @from_files = readdir(XPDF_SUBDIR); closedir(XPDF_SUBDIR); foreach $filename (@from_files) { if ($filename =~ /\.pdf/) { system("pdftotext.exe $filename"); } } exit; ***end code*** Open Source Tools for Chapter 2 ***begin code*** #!/usr/local/bin/python import os, re, string filelist = os.listdir(".") for file in filelist: if ".pdf" in file: command = "pdftotext.exe " + file os.system(command); exit ***end code*** Open Source Tools for Chapter 2 ***begin code*** #!/usr/local/bin/ruby filelist = Dir.glob("*.pdf") filelist.each do |filename| system("pdftotext.exe #{filename}") end exit ***end code*** Section 3.2. ***begin code*** #!/usr/local/bin/perl open (TEXT, "gettysbu.txt"); open(OUT, ">concordance.txt"); $/ = ""; $line = ; foreach $word (split(/[\s\n]/,$line)) { $word_location = $word_location + 1; $locations{$word} = $locations{$word} . "\,$word_location"; } foreach $word (sort keys %locations) { $locations{$word} =~ s/^[ \,]+//o; print OUT "$word $locations{$word}\n"; } exit; ***end code*** Section 3.2. ***begin code*** But 102 Four 1 God 240 It 91,160,186 Now 31 The 118,139 We 55,65 a 14,36,59,70,76,104,243 above 131 add 136 advanced. 185 ago 6 all 26 altogether 93 ***end code*** Section 3.2. ***begin code*** #!/usr/local/bin/python import re import string sentence_list = [] word_list = [] word_dict = {} format_list = [] count = 0 in_text = open('gettysbu.txt', "r") in_text_string = in_text.read() in_text_string = in_text_string.replace("\n"," ") in_text_string = in_text_string.replace(" +"," ") sentence_list = re.split(r'[\.\!\?] +(?=[A-Z])',in_text_string) for sentence in sentence_list: count = count + 1 sentence = string.lower(sentence) word_list = sentence.split() for word in word_list: if word_dict.has_key(word): word_dict[word] = word_dict[word] + ',' + str(count) else: word_dict[word] = str(count) keylist = word_dict.keys() keylist.sort() for key in keylist: print key, word_dict[key] exit ***end code*** Section 3.2. ***begin code*** #!/usr/local/bin/ruby f = File.open("gettysbu.txt") wordplace = Hash.new(""); wordarray = Array.new f.each do |line| line.downcase! line.gsub!(/[^a-z]/," ") wordarray = line.split.uniq next if wordarray == [] wordarray.each{|word| wordplace[word] = "#{wordplace[word]} #{f.lineno}"} wordarray = [] end wordplace.keys.sort.each{|key| puts "#{key} #{wordplace[key]}"} exit ***end code*** Section 3.2. ***begin code*** #!/usr/local/bin/perl open (TEXT, "concordance.TXT")||die; $line = " "; while ($line ne "") { $line = ; $line =~ s/\n/\,/o; $line =~ / /; $location_word = $`; @location_array = split(/\,/,$'); foreach $location (@location_array) { $concordance_hash{$location} = $location_word; } } $n = 1; while (exists($concordance_hash{$n})) { print $concordance_hash{$n} . " "; $n = $n + 1; } exit; ***end code*** Section 3.2. ***begin code*** Four score and seven years ago our fathers brought forth on this continent a new nation, conceived in liberty and dedicated to the proposition that all men are created equal. Now we are engaged in a great civil war, testing whether that nation or any nation so conceived and so dedicated can long endure. We are met on a great battlefield of that war. We have come to dedicate a portion of that field ***end code*** Section 3.2. ***begin code*** #!/usr/local/bin/perl open (TEXT, "concordance.TXT")||die; $line = " "; while ($line ne "") { $line = ; $line =~ s/\n/\,/o; $line =~ / /; $location_word = $`; @location_array = split(/\,/,$'); $word_hash{$location_word} = [@location_array]; foreach $location (@location_array) { $concordance_hash{$location} = $location_word; } } @word_locations_array = @{$word_hash{"who"}}; foreach $center_place (@word_locations_array) { $n = 1; print "Match.. "; while ($n < 11) { $location = $n + $center_place -5; print $concordance_hash{$location} . " "; $n = $n + 1; } print "\n"; } exit; ***end code*** Section 3.2. ***begin code*** c:\ftp>proximity_reconstruction.pl Match.. final resting-place for those who here gave their lives that Match.. men, living and dead who struggled here have consecrated it Match.. unfinished work which they who fought here have thus far ***end code*** Section 3.3. ***begin code*** #!/usr/local/bin/perl undef($/); open (TEXT, "english_lit.txt"); $line = ; while ($line =~ /\b[A-Z][a-z]+[ \n]{1}[A-Z][a-z]+\b/g) { $name = $&; $name_place = length($`); $name =~ s/\n/ /; next if ($name !~ /[A-Z][a-z]+/); $namelist{$name} = "$namelist{$name}\, $name_place"; } @namelist = sort(keys(%namelist)); foreach $name (@namelist) { print "$name $namelist{$name}\n"; } exit; ***end code*** Section 3.3. ***begin code*** Abbess Hilda , 75574, 75713 About Project , 1302966 Abraham Cowley , 409655 Abraham Cowper , 1220412 Abt Vogler , 997769, 1003750, 1008995 Academy Classics , 1212052 Adam Bede , 1073581 Adam Smith , 628914, 785572, 932577 Adelaide Procter , 1026546 Adelaide Witham , 1186897 Ado About , 315615 After Alfred , 91274 Again Beowulf , 41291, 42464 Albion Series , 96018 Aldine Edition , 942820 Aldine Poets , 211090 Aldine Series , 495306 Alexander Pope , 12315, 751310 Alexander Selkirk , 674969, 730407 Alfred Lord , 1189113 Alfred Tennyson , 13458, 1189448 Algernon Charles , 1024420, 1196870 Alice Brown , 1199231 ***end code*** Section 3.3. ***begin code*** while ($line =~ /\b[A-Z][a-z]+[ \n]{1}[A-Z][a-z]+\b/g) ***end code*** Section 3.3. ***begin code*** #!/usr/local/bin/perl @stoparray = qw(a about above according across actual actually added after afterwards again against ahead all almost alone along already also although always am among amongst amount an and another any anyhow anyone anything anyway anywhere approximately are arising around as at award away back be became because become becomes becoming been before beforehand behind being below beside besides best better between beyond birthday both bottom but by call can cannot can't certain come comes coming completely computer concerning consider considered considering consisting could couldn't cry describe despite detail did discussion do does doesn't doing done down due during each eight either eleven else elsewhere empty enough especially even ever every everyone everything everywhere except few fifteen fifty fill find fire first five followed following for former formerly forty forward found four from front full further get give given giving go had hardly has hasn't have haven't having he hence her here hereafter hereby herein hereupon hers herself him himself his honor how however hundred i if in indeed inside instead interest into is isn't it items its itself just keep largely last later latter least less let lets like likely little look looks made mainly make makes making many may maybe me meantime meanwhile meet meets might million mine miss more moreover most mostly move mr mrs much must my myself name namely near nearly neither never nevertheless next nine ninety no nobody none nonetheless nor not nothing now nowhere obtain obtained of off often on once one only onto or other others otherwise our ours ourselves out outside over overall own part per perhaps please possible possibly previously put quite rather really recent recently regarding reprinted resulted resulting same see seem seemed seeming seems seen serious seven seventy several she should shouldn't show showed shown shows side significant significantly since sincere six sixty so so-called some somehow someone something sometime sometimes somewhere still stop studies study such suggest system take taken takes taking ten than that the their theirs them themselves then there thereafter thereby therefore therein thereupon these they thick thin third thirty this those though thousand three through throughout thru thus to together too top toward towards trillion twelve twenty two under undergoing unless unlike unlikely until up upon upward us use used using various versus very via was way ways we well were weren't what whatever whats when whence whenever where whereafter whereas whereby wherein whereupon wherever whether which while whither who whoever whole whom whomever whos whose why will with within without would wouldn't yes yet you your yours yourself yourselves ); open (TEXT, "english_lit.txt"); open (OUT, ">phrase_list.txt"); undef($/); $whole_book = ; $whole_book =~ s/[\n\t]/ /g; foreach $stopword (@stoparray) { $whole_book =~ s/\-/ /g; $whole_book =~ s/ *\b$stopword\b */ vvvvv /ig; } @sentence_array = split(/[a-z]+\. +[A-Z]/, $whole_book); foreach $sentence (@sentence_array) { $sentence = lc($sentence); push(@phrasearray, split(/ *vvvvv */, $sentence)); } @phrasearray = grep($_ ne $prev && (($prev) = $_), sort(@phrasearray)); print OUT join("\n", @phrasearray); exit; ***end code*** Section 3.3. ***begin code*** shelley's "adonais," shelley's _adonais_. shelley's characters shelley's crude revolutionary doctrines. shelley's father, shelley's greater mood. shelley's influence shelley's italian life. shelley's longer poems. _adonais_ shelley's miscellaneous works, shelley's poem shelley's poem _adonais_, shelley's poetry. shelley's poetry? shelley's revolt shelley's revolutionary enthusiasm, shelley's works. . . . short descriptive passages short descriptive poems short english abstracts. short essays short halves, separated short hymns short miscellaneous poems short period short poem reflecting short poems short poems expressing short poems suggested short poems, edited short space short span short stanzas, short stories short story teller short story, short sword ***end code*** Section 3.3. ***begin code*** #!/usr/local/bin/python import re, string item_list = [] stopfile = open("stop.txt",'r') stop_list = stopfile.readlines() stopfile.close() in_text = open('list_of_sentence.txt', "r") count = 0 for line in in_text: count = count + 1 for stopword in stop_list: stopword = re.sub(r'\n', '', stopword) line = re.sub(r' *\b' + stopword + r'\b *', '\n', line) item_list.extend(line.split("\n")) item_list = sorted(set(item_list)) out_text = open('phrases.txt', "w") for item in item_list: print>>out_text, item exit ***end code*** Section 3.3. ***begin code*** #!/usr/local/bin/ruby phrase_array = [] stoparray = IO.read("stop.txt").split(/\n/) sentence_array = IO.read("list_of_sentences.txt").split(/\n/) out_text = File.open("phrases.txt", "w") sentence_array.each do |sentence| stoparray.each do |stopword| sentence.gsub!(/ *\b#{stopword}\b */, "\n") if sentence.include? stopword end phrase_array = phrase_array + sentence.split(/\n/) end out_text.puts phrase_array.sort.uniq exit ***end code*** Section 3.4. ***begin code*** adenoca of the liver adenocarcinoma arising in liver adenocarcinoma involving liver liver with cancer liver carcinoma carcinoma arising in liver cells hcc - hepatocellular carcinoma primary liver carcinoma hepatic carcinoma hepatoma hepatocarcinoma liver cell carcinoma ***end code*** Section 3.4. ***begin code*** absfile.each do |sentence| sentence.chomp! outfile.puts "\nAbstract title..." + sentence.capitalize + "." sentence_array = sentence.split length = sentence_array.size length.times do (1..sentence_array.size).each do |place_length| phrase = sentence_array.slice(0,place_length).join(" ") if literalhash.has_key?(phrase) outfile.puts "Term..." + phrase.capitalize + " " + literalhash[phrase] end end sentence_array.shift end end ***end code*** Section 3.4. ***begin code*** for line in absfile: sentence = line sentence = sentence.rstrip() sentence_array = sentence.split(" ") length = len(sentence_array) for i in range(length): for place_length in range(len(sentence_array)): last_element = place_length + 1 phrase = ' '.join(sentence_array[0:last_element]) if literalhash.has_key(phrase): print>>outfile,"Term..." + phrase + " " + literalhash[phrase] sentence_array.pop(0) exit ***end code*** Section 3.4. ***begin code*** $line = " "; while($line ne "") { $line = ; $sentence = $line; $sentence =~ s/\n//o; @sentence_array = split(/ /,$sentence); $cycles = scalar(@sentence_array); for($n=0;$n<$cycles;$n++) { for($i=0;$i suggested autosomal dominant inheritance[jb thresh.txt] 3c188dace2e7977fd6333e4d8010e181 = > mother[jb thresh.txt] 8c81b4aaf9c2009666d532da3b19d5f8 = > manifestations[jb thresh.txt] db277da2e82a4cb7e9b37c8b0c7f66f0 = > suggested[jb thresh.txt] e183376eb9cc9a301952c05b5e4e84e3 = > sons[jb thresh.txt] 22cf107be97ab08b33a62db68b4a390d = > severe[jb thresh.txt] ***end code*** Section 3.5. ***begin code*** they db277da2e82a4cb7e9b37c8b0c7f66f0 that the 8c81b4aaf9c2009666d532da3b19d5f8 were as 22cf107be97ab08b33a62db68b4a390d in the 3c188dace2e7977fd6333e4d8010e181 as in the e183376eb9cc9a301952c05b5e4e84e3 and that this 684327ec3b2f020aa3099edb177d3794. ***end code*** Section 3.5. ***begin code*** 684327ec3b2f020aa3099edb177d3794 = > suggested (autosomal dominant inheritance=C0443147)[jb thresh.txt] 3c188dace2e7977fd6333e4d8010e181 = > (mother=C0026591)[jb thresh.txt] 8c81b4aaf9c2009666d532da3b19d5f8 = > manifestations[jb thresh.txt] db277da2e82a4cb7e9b37c8b0c7f66f0 = > suggested[jb thresh.txt] e183376eb9cc9a301952c05b5e4e84e3 = > (son=C0037683)[jb thresh.txt] 22cf107be97ab08b33a62db68b4a390d = > (severe=C0205082)[jb thresh.txt] ***end code*** Section 3.5. ***begin code*** they suggested that the manifestations were as (severe=C0205082) in the (mother=C0026591) as in the (son=C0037683) and that this suggested (autosomal dominant heritance=C0443147)[jb thresh.txt] ***end code*** Open Source Tools for Chapter 3 ***begin code*** #!/usr/local/bin/python import sys, re, string print "Enter a line of text to be parsed into a word list" line = sys.stdin.readline() line = string.lower(line) line = string.rstrip(line) linearray = sorted(set(re.split(r' +', line))) for i in range(0, len(linearray)): print(linearray[i]) exit ***end code*** Open Source Tools for Chapter 3 ***begin code*** c:\ftp>words.py Enter a line of text to be parsed into a word list a way a lone a last a loved a long the riverrun, past Eve and Adam's, from swerv e of shore to bend of bay, brings us by a commodius vicus a adam's, and bay, bend brings by commodius eve from last lone long loved of past riverrun, shore swerve the to us vicus way ***end code*** Open Source Tools for Chapter 3 ***begin code*** #!/usr/local/bin/perl open(TEXT, "gettysbu.txt"); undef($/); $var = lc(); $var =~ s/\n/ /g; $var =~ s/\'s//g; $var =~ tr/a-zA-Z\'\- //cd; @words = sort(split(/ +/, $var)); @words = grep($_ ne $prev && (($prev) = $_), @words); print (join("\n",@words)); exit; ***end code*** Open Source Tools for Chapter 3 ***begin code*** #!/usr/local/bin/perl open(TEXT,"filename.txt")||die"cannot"; open(OUT,">doublets.txt")||die"cannot"; undef($/); $var = ; $var =~ s/\n/ /g; $var =~ s/\'s//g; $var =~ tr/a-zA-Z\'\- //cd; @words = split(/ +/, $var); foreach $thing (@words) { $doublet = "$oldthing $thing"; if ($doublet =~ /^[a-z]+ [a-z]+$/) { $doublethash{$doublet}=""; } $oldthing = $thing; } close TEXT; @wordarray = sort(keys(%doublethash)); print OUT join("\n",@wordarray); close OUT; exit; ***end code*** Open Source Tools for Chapter 3 ***begin code*** #!/usr/local/bin/python import anydbm, string, re in_file = open('filename.txt', "r") out_file = open('doubs.txt',"w") doubhash = {} for line in in_file: line = line.lower() line = re.sub('[.,<>?/;:"[]\{}|=+-_ ()*&^%$#@!`~1234567890]', ' ', line) hoparray = line.split() hoparray.append(" ") for i in range(len(hoparray)-1): doublet = hoparray[i] + " " + hoparray[i + 1] if doubhash.has_key(doublet): continue doubhash_match = re.search(r'[a-z]+ [a-z]+', doublet) if doubhash_match: doubhash[doublet] = "" for keys,values in sorted(doubhash.items()): out_file.write(keys + '\n') exit ***end code*** Open Source Tools for Chapter 3 ***begin code*** #!/usr/local/bin/ruby intext = File.open("filename.txt", "r") outtext = File.open("doubs.txt", "w") doubhash = Hash.new(0) line_array = Array.new(0) while record = intext.gets oldword = "" line_array = record.chomp.strip.split(/\s+/) line_array.each do |word| doublet = [oldword, word].join(" ") oldword = word next unless (doublet =~ /^[a-z]+\s[a-z]+$/) doubhash[doublet] = "" end end doubhash.each {|k,v| outtext.puts k } exit ***end code*** Open Source Tools for Chapter 3 ***begin code*** a bachelor a background a bacteremia a bacteria a bacterial a bacterium a bad a balance a balanced a banana ***end code*** Open Source Tools for Chapter 3 ***begin code*** #!/usr/local/bin/perl $text = "ngrams are ordered word sequences"; @text_list = split(" ", $text); while(scalar(@text_list) !=0) { push(@parts_list, join(" ", @text_list)); shift(@text_list); } foreach $part (@parts_list) { $previous = ""; @word_list = split(" ", $part); while(scalar(@word_list) !=0) { $ngram_list{join(" ", @word_list)} = ""; $first_word = shift(@word_list); $ngram_list{$first_word} = ""; $previous = $previous . " " . $first_word; $previous =~ s/^ //o; $ngram_list{$previous} = ""; } } print(join("\n", sort(keys(%ngram_list)))); exit; ***end code*** Open Source Tools for Chapter 3 ***begin code*** c:\ftp>ngram_list.pl are are ordered are ordered word are ordered word sequences ngrams ngrams are ngrams are ordered ngrams are ordered word ngrams are ordered word sequences ordered ordered word ordered word sequences sequences word word sequences ***end code*** Section 4.3. ***begin code*** #!/usr/local/bin/python import os os.system("cd c:/cygwin64/") os.system("imdisplay c:/some_image_file.jpg") exit ***end code*** Section 4.3. ***begin code*** #!/usr/bin/perl opendir(FILES, ".") || die ("Unable to open directory"); @in_files = readdir(FILES); closedir(FILES); foreach $filename (@in_files) { if ($filename =~ /([a-z0-9\_]+)(\.[gifbmpjpgn]{3}) *$/i) { $out_file = $1 . "_bw" . $2; system("convert ${filename} -set colorspace Gray -separate -average ${out_file}"); } } exit; ***end code*** Section 4.3. ***begin code*** #!/usr/local/bin/python import os, re, string filelist = os.listdir(".") for file in filelist: if ".jpg" in file or ".bmp" in file or ".gif" in file or ".png" in file: outfile = "bw_" + file command = "convert " + file + " -set colorspace Gray -separate -average " + outfile os.system(command) exit ***end code*** Section 4.3. ***begin code*** #!/usr/local/bin/ruby filelist = Dir.glob("*.*") filelist.each do |file| if file =~ /([a-z0-9\_]+)(\.[gifbmpjpgn]{3}) *$/i out_file = $1 + "_bw" + $2 system("convert " + file + " -set colorspace Gray -separate -average " + out_file) end end exit ***end code*** Section 4.3. ***begin code*** #!/usr/local/bin/ruby filelist = Dir.glob("*.*") filelist.each do |file| if file =~ /([a-z0-9\_]+)(\.[gifbmpjpgn]{3}) *$/i out_file = $1 + "_resized" + $2 system("convert " + file + " -resize 325x500! " + out_file) end end exit ***end code*** Section 4.3. ***begin code*** c:\ftp>identify -verbose 071.jpg > 071_header.txt ***end code*** Section 4.3. ***begin code*** c:\ftp>convert eqn.jpg -set comment "hello world yet again" eqn.jpg ***end code*** Section 4.3. ***begin code*** c:\ftp>identify -verbose -format "%c" eqn.jpg hello world yet again ***end code*** Section 4.3. ***begin code*** c:\ftp>convert eqn.jpg -set comment @temp.txt eqn.jpg ***end code*** Section 4.4. ***begin code*** compare -metric phash image1 image2 diffimage ***end code*** Section 4.4. ***begin code*** c:\photos\rocks>compare -metric phash Adamite_wiki.jpg autunite.jpg diffimage 297.975 ***end code*** Section 4.4. ***begin code*** c:\photos\rocks>compare -metric phash Crassostrea_gigas_oyster_2.jpg Crassostrea_gigas_oyster.jpg diffimage 99.2882 ***end code*** Section 4.4. ***begin code*** c:\photos\rocks>compare -metric phash autunite.jpg autunite.jpg diffimage 0 ***end code*** Section 4.4. ***begin code*** $ chmod +x phashconvert ***end code*** Section 4.4. ***begin code*** $ phashconvert orca.jpg 002400630193027003310533030506560695110004470870062411000020-05301770056032700260293001807380177041800820604004000250022018501830337033002980297077007530422042706150611 $ phashconvert tar1.jpg 0034-0190148012303710039035800150751012804680115072900430026-03701310014030901380298011806310282040201690608025100240028012801360305032702940316062206660395042106000644 ***end code*** Section 4.4. ***begin code*** #!/usr/local/bin/ruby freq = Hash.new(0) my_string = "Peter Piper picked a peck of pickled peppers. A peck of pickled peppers Peter Piper picked. If Peter Piper picked a peck of pickled peppers, Where is the peck of pickled peppers that Peter Piper picked?" my_string.downcase.scan(/\w+/){|word| freq[word] = freq[word]+1} freq.sort_by {|k, v| v}.reverse.each {|k,v| printf "%-10.10s %0.2u\n", k, v} exit ***end code*** Section 4.4. ***begin code*** output: peter 04 piper 04 picked 04 peck 04 of 04 peppers 04 pickled 04 a 03 if 01 where 01 is 01 the 01 that 01 ***end code*** Section 4.4. ***begin code*** #!/usr/local/bin/perl $my_string = <<"EOF"; Peter Piper picked a peck of pickled peppers. A peck of pickled peppers Peter Piper picked. If Peter Piper picked a peck of pickled peppers, Where is the peck of pickled peppers that Peter Piper picked? EOF $my_string =~ s/\n\.\?/ /g; $my_string = lc($my_string); while ($my_string =~ /\b([a-z]+)\b/g) { $freq{$1}++; } while ((my $key, my $value) = each(%freq)) { $value = "000000" . $value; $value = substr($value,-6,6); push (@termarray, "$value $key"); } print join("\n", (reverse (sort (@termarray)))); exit; ***end code*** Section 4.4. ***begin code*** #!/usr/local/bin/python import re import string word_list = [] freq_list = [] format_list = [] freq = {} my_string = "Peter Piper picked a peck of pickled peppers. A peck of pickled peppers Peter Piper picked. If Peter Piper picked a peck of pickled peppers, Where is the peck of pickled peppers that Peter Piper picked?".lower() word_list = re.findall(r'(\b[a-z]{1,}\b)', my_string) for item in word_list: count = freq.get(item,0) freq[item] = count + 1 for key, value in freq.iteritems(): value = "000000" + str(value) value = value[-6:] format_list += [value + " " + key] format_list = reversed(sorted(format_list)) print("\n".join(format_list)) exit ***end code*** Section 4.4. ***begin code*** set xtics border in scale 1,0.5 nomirror rotate by -90 offset character 0,0,0; set term png size 1300,800; set output 'c:\out.png'; unset key; set xrange [0:80]; set style fill solid; plot 'c:\gettysb.dat' using 2:xticlabels(1) with histogram; ***end code*** Section 4.4. ***begin code*** c:\Program Files (x86)\gnuplot\bin>gnuplot.exe c:\gettysburg.gp ***end code*** Section 4.5. ***begin code*** irb(main):001:0> Math.sqrt((5**2) * 10000) => 500.0 ***end code*** Open Source Tools for Chapter 4 ***begin code*** gnuplot> load 'c:\ftp\peter.gp' ***end code*** Open Source Tools for Chapter 4 ***begin code*** #!/usr/local/bin/python import numpy import matplotlib.pyplot as plt cent_array = numpy.linspace(1,50,2000) sine_array = map(lambda x: numpy.sin(x), cent_array) plt.plot(cent_array, sine_array) plt.show() exit ***end code*** Open Source Tools for Chapter 4 ***begin code*** #!/usr/local/bin/python import numpy import matplotlib.pyplot as plt cent_array = numpy.linspace(1,50,2000) mixed_sine_cosine_array = \ map(lambda x: (0.1*numpy.sin(0.7*x)) + (0.7*numpy.cos(0.1*x)), cent_array) plt.plot(cent_array, mixed_sine_cosine_array) plt.show() exit ***end code*** Open Source Tools for Chapter 4 ***begin code*** #!/usr/bin/python import numpy from numpy.linalg import inv a = numpy.array([[1,4,6],[9,15,55],[62,-5, 4]]) print(a) print(inv(a)) c = numpy.dot(a, inv(a)) print(numpy.round_(c)) exit ***end code*** Open Source Tools for Chapter 4 ***begin code*** c:\ftp\py>numpy_dot.py [[ 1 4 6] [ 9 15 55] [62 -5 4]] [[ 4.19746899e-02 -5.76368876e-03 1.62886856e-02] [ 4.22754041e-01 -4.61095101e-02 -1.25297582e-04] [ -1.22165142e-01 3.17002882e-02 -2.63124922e-03]] [[ 1. 0. 0.] [ 0. 1. 0.] [ 0. 0. 1.]] ***end code*** Open Source Tools for Chapter 4 ***begin code*** c:\ftp>convert mtor_pathway.svg mtor_pathway.jpg ***end code*** Open Source Tools for Chapter 4 ***begin code*** convert -size 600x800 image.jpg -thumbnail 120x160 image_thumb.gif ***end code*** Open Source Tools for Chapter 4 ***begin code*** C:\ftp>convert original.jpg -contrast -contrast result.png ***end code*** Open Source Tools for Chapter 4 ***begin code*** c:\ftp>convert -comment @words.txt tar1.jpg tar1.jpg ***end code*** Open Source Tools for Chapter 4 ***begin code*** c:\ftp>identify -verbose tar1.jpg ***end code*** Open Source Tools for Chapter 4 ***begin code*** #!/usr/local/bin/python import sys, os, re filelist = os.listdir(".") pattern = re.compile(".jpg$") for filename in filelist: if pattern.search(filename): out_filename = pattern.sub('_small.jpg', filename) print(out_filename) cmdstring = "convert " + filename + " -resize 400x267! " + out_filename os.system(cmdstring) else: continue exit ***end code*** Open Source Tools for Chapter 4 ***begin code*** #!/usr/local/bin/perl $newdir = "c\:\\ftp\\factth"; opendir (MYDIR, $newdir) || die ("Can't open directory"); chdir ($newdir); while ($file = readdir (MYDIR)) { next if (-d $file); next if ($file eq "." || $file eq ".."); system("convert $file -resize 30% $file"); } closedir (MYFILE); exit; ***end code*** Open Source Tools for Chapter 4 ***begin code*** #!/usr/local/bin/python import matplotlib.pyplot as plt plt.text(0.1, 0.5, r'$n = (Z_{\alpha/2} \sigma / E)^2$', fontsize=40) plt.show() exit ***end code*** Open Source Tools for Chapter 4 ***begin code*** #!/usr/local/bin/python from random import randint outfile = open ("random_1.raw", "wb") for n in range(160000): c = chr(randint(0,255)) outfile.write(c) exit ***end code*** Open Source Tools for Chapter 4 ***begin code*** c:\ftp>convert -size 400x400 -depth 8 gray:random_1.raw random_1.bmp ***end code*** Open Source Tools for Chapter 4 ***begin code*** #!/usr/local/bin/python import gzip import os first_in = open('random_1.raw', 'rb') first_out = gzip.open('random_1.gz', 'wb') first_out.writelines(first_in) first_out.close() first_in.close() random_1_size = os.path.getsize('random_1.gz') second_in = open('random_2.raw', 'rb') second_out = gzip.open('random_2.gz', 'wb') second_out.writelines(second_in) second_out.close() second_in.close() random_2_size = os.path.getsize('random_2.gz') os.system("c:/cygwin64/bin/cat c:/ftp/random_1.raw \ c:/ftp/random_2.raw > c:/ftp/random_cat.raw") cat_in = open('random_cat.raw', 'rb') cat_out = gzip.open('random_cat.gz', 'wb') cat_out.writelines(cat_in) cat_out.close() cat_in.close() random_cat_size = os.path.getsize('random_cat.gz') dist = float(random_cat_size - random_1_size)/float(random_2_size) print(dist) exit ***end code*** Open Source Tools for Chapter 4 ***begin code*** c:\ftp>norm_comp.py 0.999843825159 ***end code*** Open Source Tools for Chapter 4 ***begin code*** #!/usr/bin/python from scipy.stats.stats import pearsonr a = [1,2,3,4] b = [2,4,6,8] print pearsonr(a,b) exit ***end code*** Open Source Tools for Chapter 4 ***begin code*** c:\ftp\py>sci_pearson.py (0.99339926779878274, 0.073186395040328034) ***end code*** Open Source Tools for Chapter 4 ***begin code*** #!/usr/bin/python from scipy.stats.stats import pearsonr a = [1,4,6,9,15,55,62,-5] b = [-2,-8,-9,-12,-80,14,15,2] print pearsonr(a,b) exit ***end code*** Open Source Tools for Chapter 4 ***begin code*** c:\ftp\py>sci_pearson.py (0.32893766587262174, 0.42628658412101167) ***end code*** Open Source Tools for Chapter 4 ***begin code*** a = [1,4,6,9,15,55,62,-5] b = [-2,-8,-9,-12,-80,14,15,2] c = [2,8,12,18,30,110,124,-10] ***end code*** Open Source Tools for Chapter 4 ***begin code*** #!/usr/bin/python from __future__ import division import numpy from numpy import linalg a = [1,4,6,9,15,55,62,-5] b = [-2,-8,-9,-12,-80,14,15,2] c = [2,8,12,18,30,110,124,-10] a_length = linalg.norm(a) b_length = linalg.norm(b) c_length = linalg.norm(c) print "Vector a is ", (a) print "Vector b is ", (b) print "Vector c is ", (c) print "Length of a is ", (a_length) print "Length of b is ", (b_length) print "Length of c is ", (c_length) print "Dot of a and b is ", (numpy.dot(a,b)) print "Dot of b and a is ", (numpy.dot(b,a)) print "Dot of a and a is ", (numpy.dot(a,a)) print "Dot of b and b is ", (numpy.dot(b,b)) print "Dot of a and c is ", (numpy.dot(a,c)) print "Dot of c and b is ", (numpy.dot(c,b)) print "Normalized dot of a and b is ", (numpy.dot(a,b) / (a_length * b_length)) print "Normalized dot a and a is ", (numpy.dot(a,a) / (a_length * a_length)) print "Normalized dot of b and b is ", (numpy.dot(b,b) / (b_length * b_length)) print "Normalized dot of a and c is ", (numpy.dot(a,c) / (a_length * c_length)) print "Normalized dot of b and c is ", (numpy.dot(b,c) / (b_length * c_length)) exit ***end code*** Open Source Tools for Chapter 4 ***begin code*** c:\ftp>dot_big.py Vector a is [1, 4, 6, 9, 15, 55, 62, -5] Vector b is [-2, -8, -9, -12, -80, 14, 15, 2] Vector c is [2, 8, 12, 18, 30, 110, 124, -10] Length of a is 85.1645466142 Length of b is 84.36824047 Length of c is 170.329093228 Dot of a and b is 294 Dot of b and a is 294 Dot of a and a is 7253 Dot of b and b is 7118 Dot of a and c is 14506 Dot of c and b is 588 Normalized dot of a and b is 0.0409175385118 Normalized dot a and a is 1.0 Normalized dot of b and b is 1.0 Normalized dot of a and c is 1.0 Normalized dot of b and c is 0.0409175385118 ***end code*** Section 5.1. ***begin code*** $ uuidgen.exe 312e60c9-3d00-4e3f-a013-0d6cb1c9a9fe $ uuidgen.exe 822df73c-8e54-45b5-9632-e2676d178664 $ uuidgen.exe 8f8633e1-8161-4364-9e98-fdf37205df2f $ uuidgen.exe 83951b71-1e5e-4c56-bd28-c0c45f52cb8a $ uuidgen -t e6325fb6-5c65-11e5-b0e1-0ceee6e0b993 $ uuidgen -r 5d74e36a-4ccb-42f7-9223-84eed03291f9 ***end code*** Section 5.1. ***begin code*** c:\ftp>identify -verbose -format "%#" eqn.jpg ***end code*** Section 5.1. ***begin code*** 219e41b4c761e4bb04fbd67f71cc84cd6ae53a26639d4bf33155a5f62ee36e33 ***end code*** Section 5.1. ***begin code*** c:\ftp>convert eqn.jpg -set comment "I'm modifying myself" eqn.jpg ***end code*** Section 5.1. ***begin code*** c:\ftp>identify -verbose -format "%c" eqn.jpg ***end code*** Section 5.1. ***begin code*** I'm modifying myself ***end code*** Section 5.1. ***begin code*** c:\ftp>identify -verbose -format "%#" eqn.jpg ***end code*** Section 5.1. ***begin code*** cb448260d6eeeb2e9f2dcb929fa421b474021584e266d486a6190067a278639f ***end code*** Section 5.1. ***begin code*** #!/usr/local/bin/python import sys, os, uuid my_id = "" + str(uuid.uuid4()) + "" in_command = "convert eqn.jpg -set comment \"" + my_id + "\" eqn.jpg" os.system(in_command) out_command = "identify -verbose -format \"%c\" eq2.jpg" print ("\nHere's the unique identifier:") os.system(out_command) print ("\nHere's the unique authenticator:") os.system("identify -verbose -format \"%#\" eqn.jpg") os.system("convert eqn.jpg -resize 325x500! eqn.jpg") print ("\nHere's the new authenticator:") os.system("identify -verbose -format \"%#\" eqn.jpg") print ("\nHere's the unique identifier:") os.system(out_command) exit ***end code*** Section 5.1. ***begin code*** Here's the unique identifier: c94f679f-7acd-4216-a464-eb051ab57547 Here's the unique authenticator: 3529d28f97661b401d9ce6d9925a2dadb46c26b7350d94fff5585d7860886781 Here's the new authenticator: 7b45485ca7fca87f5b78e87b9392946b3e1895dab362d2ca5b13a0e3bc136e48 Here's the unique identifier: c94f679f-7acd-4216-a464-eb051ab57547 ***end code*** Section 5.1. ***begin code*** my_id = "" + str(uuid.uuid4()) + "" in_command = "convert eqn.jpg -set comment \"" + my_id + "\" eqn.jpg" os.system(in_command) ***end code*** Section 5.1. ***begin code*** Here's the unique identifier: c94f679f-7acd-4216-a464-eb051ab57547 Here's the unique authenticator: 3529d28f97661b401d9ce6d9925a2dadb46c26b7350d94fff5585d7860886781 ***end code*** Section 5.1. ***begin code*** Here's the new authenticator: 7b45485ca7fca87f5b78e87b9392946b3e1895dab362d2ca5b13a0e3bc136e48 Here's the unique identifier: c94f679f-7acd-4216-a464-eb051ab57547 ***end code*** Section 5.3. ***begin code*** 75898039563441 name G. Willikers 75898039563441 gender male 75898039563441 is_a_class_member cowboy 75898039563441 age 35 94590439540089 name Hopalong Tagalong 94590439540089 is_a_class_member cowboy 29847575938125 calendar:date February 4, 1986 57839109275632 social:date Jack and Jill 83654560466294 social:date Pyramus and Thisbe 83654560466294 calendar:date June 16, 1904 98495efc object_name Andy Muzeack 98495efc instance_of Homo sapiens 98495efc dob 1 January, 2001 98495efc glucose_at_time 87, 02-12-2014 17:33:09 ***end code*** Section 5.3. ***begin code*** 75898039563441 gender male 75898039563441 is_a_class_member cowboy 94590439540089 is_a_class_member cowboy 29847575938125 calendar:date February 4, 1986 83654560466294 calendar:date June 16, 1904 98495efc instance_of Homo sapiens 98495efc glucose_at_time 87, 02-12-2014 17:33:09 ***end code*** Section 5.3. ***begin code*** 75898039563441 gender male 75898039563441 is_a_class_member cowboy 75898039563441 mental_status borderline personality disorder 75898039563441 criminal_record multiple assaults 75898039563441 chief_complaint kicked in head by cow ***end code*** Section 5.3. ***begin code*** 82030201856150 gender male 82030201856150 is_a_class_member cowboy 44934938405062 is_a_class_member cowboy 65840231656302 calendar:date February 4, 1986 76206674367326 calendar:date June 16, 1904 7392g2s1 instance_of Homo sapiens 7392g2s1 glucose_at_time 87, 02-12-2014 17:33:09 ***end code*** Section 5.3. ***begin code*** 75898039563441 -> 82030201856150 94590439540089 -> 44934938405062 29847575938125 -> 65840231656302 83654560466294 -> 76206674367326 98495efc -> 7392g2s1 ***end code*** Section 5.4. ***begin code*** #!/usr/local/bin/ruby f = File.open "doubdb.txt" outf = File.open("scrub.out", "w") doubhash = Hash.new while line = f.gets line = chomp] doubhash[line] = " " end f.close puts "What would you like to scrub?" line = gets.chomp linearray = line.split arraysize = linearray.length - 2 lastword = "*" for arrayword in (0 .. arraysize) phrase = linearray[arrayword] + " " + linearray[arrayword+1] if doubhash.key?(phrase) print " " + linearray[arrayword] lastword = " " + linearray[arrayword+1] else print lastword lastword = " *" end if arrayword == arraysize print lastword end end[jb ruby.txt ***end code*** Section 5.4. ***begin code*** Basal cell carcinoma, margins involved Scrubbed text.... basal cell carcinoma margins involved Mr Brown has a basal cell carcinoma Scrubbed text.... * * has a basal cell carcinoma Mr. Brown was born on Tuesday, March 14, 1985 Scrubbed text.... * * * * * * * * *] The doctor killed the patient Scrubbed text.... * * * * *[jb ruby.txt] ***end code*** Section 5.5. ***begin code*** #!/usr/bin/python import codecs print('abCdeFgHijKlM') output = codecs.encode('abCdeFgHijKlM', 'rot_13') print output print(codecs.encode(output, 'rot_13')) print('hello world') output = codecs.encode('hello world', 'rot_13') print output print(codecs.encode(output, 'rot_13')) exit ***end code*** Section 5.5. ***begin code*** c:\ftp>rot_13.py abCdeFgHijKlM noPqrStUvwXyZ abCdeFgHijKlM hello world uryyb jbeyq hello world ***end code*** Section 5.5. ***begin code*** #!/usr/local/bin/perl $text = "abCdeFgHijKlM and hello world"; $text =~ tr/A-Za-z/N-ZA-Mn-za-m/; print $text; print "\n"; $text =~ tr/A-Za-z/N-ZA-Mn-za-m/; print $text; exit; ***end code*** Section 5.5. ***begin code*** c:\ftp>rot_13.pl noPqrStUvwXyZ naq uryyb jbeyq abCdeFgHijKlM and hello world ***end code*** Section 5.5. ***begin code*** #!/usr/bin/python import sys, os, re, codecs current_directory = os.getcwd() filelist = os.listdir(current_directory) pattern = re.compile(".txt$") ascii_characters = re.compile("[a-zA-Z0-9 _]") for in_file_name in filelist: if pattern.search(in_file_name): out_file_name = pattern.sub('.rot', in_file_name) print(out_file_name) out_file_holder = open(out_file_name,'w') with open(in_file_name) as in_file_holder: while True: character = in_file_holder.read(1) if not character: break if not ascii_characters.search(character): out_file_holder.write(character) else: rot_13_of_character = codecs.encode(character, 'rot_13') out_file_holder.write(rot_13_of_character) else: continue exit ***end code*** Section 5.5. ***begin code*** #!/usr/local/bin/python import sys, os, re filelist = ['diener.txt','simplify.txt','re-ana.txt', 'phenocop.txt', 'mystery.txt','disaster.txt', 'factnote.txt', 'perlbig.txt', 'referen.txt', 'create.txt', 'exploreo.txt'] pattern = re.compile("txt") for filename in filelist: out_filename = pattern.sub('enc', filename) out_filename = "f:\\" + out_filename print(out_filename) cmdstring = "openssl aes128 -in " + filename + " -out " + out_filename + " -pass pass:z4u7w28" os.system(cmdstring) exit ***end code*** Section 5.6. ***begin code*** Data object -> Newspaper headlines: "Dewey Defeats Truman" timestamp: November 3,1948, 6:00 AM "Dewey Defeats Truman" (modification) timestamp: November 3,1948, 10:00 AM "Truman Defeats Dewey" timestamp: November 3, 1948, 10:01 AM ***end code*** Section 5.6. ***begin code*** c:\ftp>perl -e print(time()) 1442353564 ***end code*** Section 5.6. ***begin code*** c:\ftp>python >>> import time >>> print(time.time()) 1442353742.456994 ***end code*** Section 5.6. ***begin code*** c:\ftp>irb irb(main):001:0> "%10.9f" % Time.now.to_f => "1442354071.895107031" ***end code*** Section 5.6. ***begin code*** #!/usr/bin/perl ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = gmtime(); $year = $year + 1900; $mon = $mon + 1; print "Americanized GMT date is: $mon\/$mday\/$year\n"; exit; ***end code*** Section 5.6. ***begin code*** c:\ftp>gmt.pl Americanized GMT date is: 9/16/2015 ***end code*** Section 5.6. ***begin code*** #!/usr/bin/perl ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = gmtime(); $year = $year + 1900; $mon = substr(("000" . ($mon+1)), -2, 2); $mday = substr(("000" . $mday), -2, 2); $hour = substr(("000" . $hour), -2, 2); $min = substr(("000" . $min), -2, 2); $sec = substr(("000" . $sec), -2, 2); print "Americanized time is: $mday\/$wday\/$year\n"; print "ISO8601 time is:$year\-$mon\-$mday $hour\:$min\:$sec\n"; print "ISO8601 time is:$year\-$mon\-${mday}T$hour\:$min\:${sec}Z (alternate form)"; exit; ***end code*** Section 5.6. ***begin code*** c:\ftp>format_time.pl Americanized time is: 16/3/2015 ISO8601 time is:2015-09-16 12:31:41 ISO8601 time is:2015-09-16T12:31:41Z (alternate form) ***end code*** Section 5.6. ***begin code*** #!/usr/bin/python import time, datetime timenow = time.time() print(datetime.datetime.fromtimestamp(timenow).strftime('%Y-%m-%d %H:%M:%S')) exit ***end code*** Section 5.6. ***begin code*** c:\ftp>format_time.py 2015-09-16 07:44:09 ***end code*** Open Source Tools for Chapter 5 ***begin code*** irb(main):001:0> (1..10).each{puts rand()} ***end code*** Open Source Tools for Chapter 5 ***begin code*** 0.6210719721375545 0.8275281308969118 0.5221605121682973 0.4579032986235061 0.3897775291626894 0.1859092284180266 0.9087949176336569 0.44303624386264195 0.514384506264992 0.037523700988150055 ***end code*** Open Source Tools for Chapter 5 ***begin code*** import random for iterations in range(10): print random.uniform(0,1) ***end code*** Open Source Tools for Chapter 5 ***begin code*** for (0..9) { print rand() . "\n"; } ***end code*** Open Source Tools for Chapter 5 ***begin code*** #!/usr/bin/python import random filename = [0]*12 filename = map(lambda x: x is "" or chr(int(random.uniform(0,25) + 97)), filename) print ''.join(filename[0:8]) + "." + ''.join(filename[9:12]) exit ***end code*** Open Source Tools for Chapter 5 ***begin code*** c:\ftp\py>random_filenames.py tjqimddr.mjb ***end code*** Open Source Tools for Chapter 5 ***begin code*** #!/usr/local/bin/python from random import randint outfile = open ("random_1.raw", "wb") for n in range(160000): c = chr(randint(0,255)) outfile.write(c) exit ***end code*** Open Source Tools for Chapter 5 ***begin code*** 4c108407-0570-4afb-9463-2831bcc6e4a4[jb meta.txt] ***end code*** Open Source Tools for Chapter 5 ***begin code*** c:\cygwin64\bin>uuidgen.exe ***end code*** Open Source Tools for Chapter 5 ***begin code*** 9ee64643-2ff2-4cd1-ad31-ab59f933a276 ***end code*** Open Source Tools for Chapter 5 ***begin code*** #!/usr/bin/perl system("c\:\\cygwin64\\bin\\uuidgen.exe"); exit; ***end code*** Open Source Tools for Chapter 5 ***begin code*** #!/usr/local/bin/python import uuid print uuid.uuid4() exit ***end code*** Open Source Tools for Chapter 5 ***begin code*** gem install guid ***end code*** Open Source Tools for Chapter 5 ***begin code*** #!/usr/local/bin/ruby require 'guid' puts Guid.new exit ***end code*** Open Source Tools for Chapter 5 ***begin code*** c:\>openssl aes128 -in public.txt -out secret.aes -pass pass:z4u7w28" ***end code*** Open Source Tools for Chapter 5 ***begin code*** #!/usr/bin/perl system("openssl aes128 -in public.txt -out secret.aes -pass pass:z4u7w28"); exit; ***end code*** Open Source Tools for Chapter 5 ***begin code*** #!/usr/bin/perl chdir "c\:\\ftp\\openssl\-1\.0\.1l\-x64\_86\-win64"; system("openssl des -in c\:\\ftp\\public.txt -out c\:\\ftp\\secret.des -pass pass:test123"); exit; ***end code*** Open Source Tools for Chapter 5 ***begin code*** #!/usr/bin/perl system("openssl aes128 -d -in secret.aes -out decrypted.txt -pass pass:z4u7w28"); exit; ***end code*** Open Source Tools for Chapter 5 ***begin code*** #!/usr/bin/perl chdir "c\:\\ftp\\openssl\-1\.0\.1l\-x64\_86\-win64"; system("openssl des -d -in c\:\\ftp\\secret.des -out c\:\\ftp\\secret.txt -pass pass:test123"); exit; ***end code*** Open Source Tools for Chapter 5 ***begin code*** c:\ftp>c:\cygwin64\bin\md5sum.exe dash.png ***end code*** Open Source Tools for Chapter 5 ***begin code*** db50dc33800904ab5f4ac90597d7b4ea *dash.png ***end code*** Open Source Tools for Chapter 5 ***begin code*** #!/usr/local/bin/python import sys, os os.system("c:/cygwin64/bin/md5sum.exe dash.png") exit ***end code*** Open Source Tools for Chapter 5 ***begin code*** #!/usr/local/bin/ruby system("c:/cygwin64/bin/md5sum.exe dash.png") exit ***end code*** Open Source Tools for Chapter 5 ***begin code*** #!/usr/local/bin/perl system("c:/cygwin64/bin/md5sum.exe dash.png"); exit; ***end code*** Open Source Tools for Chapter 5 ***begin code*** db50dc33800904ab5f4ac90597d7b4ea *dash.png ***end code*** Open Source Tools for Chapter 5 ***begin code*** #!/usr/bin/perl system("openssl dgst -sha public.txt >hash.sha"); exit; ***end code*** Open Source Tools for Chapter 5 ***begin code*** SHA(public.txt)= af2f12663145770ac0cbd260e69675af6ac26417 ***end code*** Open Source Tools for Chapter 5 ***begin code*** #!/usr/local/bin/ruby system("openssl dgst -sha public.txt >hash.sha") exit ***end code*** Open Source Tools for Chapter 5 ***begin code*** SHA(public.txt)= af2f12663145770ac0cbd260e69675af6ac26417 ***end code*** Open Source Tools for Chapter 5 ***begin code*** #!/usr/local/bin/python import os filelist = os.listdir(".") outfile = open ("hashes_collect.txt", "w") for filename in filelist: cmdstring = "openssl dgst -sha %s" % (filename) cmdstring = cmdstring + "> hashes.sha" os.system(cmdstring) infile = open ("hashes.sha", "r") getline = infile.readline() infile.close() outfile.write(getline) outfile.close() exit ***end code*** Open Source Tools for Chapter 5 ***begin code*** SHA(bunzip2.TXT)= 0547d31d7c675ae2239067611e6309dc8cb7e7db SHA(googlez.TXT)= f96cea198ad1dd5617ac084a3d92c6107708c0ef SHA(gunzipe.TXT)= d275bd87933f2322a97c70035aa2aa5f4c6088ac SHA(gzip.TXT)= 0b323cb4555c8996c8100f8ad8259eec2538821b SHA(hashes.sha)= f96cea198ad1dd5617ac084a3d92c6107708c0ef SHA(hashes_collect.txt)= f96cea198ad1dd5617ac084a3d92c6107708c0ef SHA(hash_dir.py)= 1db37524e54de40ff723fbc7e3ba20b18e651d48 SHA(JHSPZIP.TXT)= 32424d0d1fe75bedd5680205fbbc63bff4bb783a SHA(libeay32.dll)= 647fae7916e8c4c45d0002fd6d2fc9c6877de085 SHA(mortzip.TXT)= 3168f52511c8289db7657b637156063c0e8c5646 SHA(Nonlinear_Science_FAQ.txt)= 6316b8531712ca2f0b192c1f662dbde446f958d9 SHA(openssl.exe)= 1cf6af2d3f720e0959d0ce49d6e4dad7a58092e8 SHA(pesonalized_blog.txt)= 25d208163e7924b8c10c7e9732d29383d61a22f1 SHA(ssleay32.dll)= 4889930b67feef5765d3aef3d1752db10ebced8f ***end code*** Open Source Tools for Chapter 5 ***begin code*** c:\ftp\steghide\steghide>steghide embed -cf c:\ftp\simplify\berman_author_photo. jpg -ef c:\ftp\berman_author_bio.txt -p hideme ***end code*** Open Source Tools for Chapter 5 ***begin code*** #!/usr/local/bin/python import os command_string = "steghide embed -cf c:/ftp/simplify/berman_author_photo.jpg -ef c:/ftp/berman_author_bio.txt -p hideme" os.system(command_string) exit ***end code*** Section 6.1. ***begin code*** "Jules Berman" "blood glucose level" "85" "Mary Smith" "blood glucose level" "90" "Samuel Rice" "blood glucose level" "200" "Jules Berman" "eye color" "brown" "Mary Smith" "eye color" "blue" "Samuel Rice" "eye color" "green" ***end code*** Section 6.1. ***begin code*** "Juan Valdez" "hat size" "8" "Jules Berman" "hat size" "9" "Homer Simpson" "hat size" "9" "Homer Simpson" "hat_type" "bowler" ***end code*** Section 6.1. ***begin code*** "Jules Berman" "blood glucose level" "85" "Jules Berman" "eye color" "brown" "Jules Berman" "hat size" "9" ***end code*** Section 6.1. ***begin code*** 75898039563441 name G. Willikers 75898039563441 gender male[jb big_data.txt] ***end code*** Section 6.1. ***begin code*** 75898039563441 age 35 75898039563441 is_a_class_member cowboy 94590439540089 name Hopalong Tagalong 94590439540089 is_a_class_member cowboy[jb big_data.txt] ***end code*** Section 6.1. ***begin code*** 75898039563441 name G. Willikers 75898039563441 gender male 75898039563441 is_a_class_member cowboy 75898039563441 age 35 94590439540089 name Hopalong Tagalong 94590439540089 is_a_class_member cowboy[jb big_data.txt] ***end code*** Section 6.1. ***begin code*** 29847575938125 calendar:date February 4, 1986 83654560466294 calendar:date June 16, 1904[jb big_data.txt] ***end code*** Section 6.1. ***begin code*** 57839109275632 social:date Jack and Jill 83654560466294 social:date Pyramus and Thisbe[jb big_data.txt] ***end code*** Section 6.1. ***begin code*** 29847575938125 calendar:date February 4, 1986 57839109275632 social:date Jack and Jill 83654560466294 social:date Pyramus and Thisbe 83654560466294 calendar:date June 16, 1904[jb big_data.txt] ***end code*** Section 6.4. ***begin code*** while() { print OUT "$namehash{$id_name}\n"; $id_name = $parenthash{$id_name}; last if ($namehash{$id_name} eq "root"); } ***end code*** Section 6.4. ***begin code*** for i in range(30): if namehash.has_key(id_name): print>>outtext, namehash[id_name] if parenthash.has_key(id_name): id_name = parenthash[id_name] ***end code*** Section 6.4. ***begin code*** (1..30).each do outtext.puts(namehash[id_name]) id_name = parenthash[id_name] break if namehash[id_name].nil? ***end code*** Section 6.5. ***begin code*** Batman has_component leg ***end code*** Section 6.5. ***begin code*** Batman (a unique instance of Class Human), has_component (a metadata property of Class Human) leg (the data described by the "has_component" property). ***end code*** Open Source Tools for Chapter 6 ***begin code*** Normal Lung ***end code*** Open Source Tools for Chapter 6 ***begin code*** Normal Lung Bill Moore 2006-06-28 ***end code*** Open Source Tools for Chapter 6 ***begin code*** Normal Lung Bill Moore 2006-06-28 ***end code*** Open Source Tools for Chapter 6 ***begin code*** Normal Lung Bill Moore 2006-06-28 ***end code*** Open Source Tools for Chapter 6 ***begin code*** @prefix : . @prefix rdf: . :Baltimore_Hospital_Center rdf:type "Hospital". :Baltimore_Hospital_Center_4357 rdf:type "Unique_medical_identifier". :Baltimore_Hospital_Center_4357 :patient_name "Sam_Someone". :Baltimore_Hospital_Center_4357 :surgical_pathology_specimen "S3456_2001". :S_3456_2001 rdf:type "Surgical_pathology_specimen". :S_3456_2001 :image . :S_3456_2001 :log_in_date "2001-08-15". :S_3456_2001 :clinical_history "30_years_oral_tobacco_use". rdf:type "Medical_image". :specimen "2". :block "3". :format "jpeg". :width "524_pixels". :height "429_pixels". ***end code*** Open Source Tools for Chapter 6 ***begin code*** rdf:type "Medical_image". ***end code*** Open Source Tools for Chapter 6 ***begin code*** @prefix rdf: @prefix rdfs: @base :Homo instance_of rdfs:Class. :HomoSapiens instance_of rdfs:Class; rdfs:subClassOf :Homo. ***end code*** Open Source Tools for Chapter 6 ***begin code*** :Batman instanceOf :HomoSapiens. ***end code*** Open Source Tools for Chapter 6 ***begin code*** c:\ftp>pip install rdflib ***end code*** Open Source Tools for Chapter 6 ***begin code*** Bob Dylan USA Columbia 10.90 1985 Bonnie Tyler UK CBS Records 9.90 1988 ***end code*** Open Source Tools for Chapter 6 ***begin code*** #!/usr/bin/python import rdflib g=rdflib.Graph() g.load('rdf_example.xml') for subject,predicate,object in g: print "Identified subject -", subject print "Metadata -", predicate print "Data -", object print exit ***end code*** Open Source Tools for Chapter 6 ***begin code*** c:\ftp>rdf_parse.py No handlers could be found for logger "rdflib.term" Identified subject - http://www.recshop.fake/cd/Hide your heart Metadata - http://www.recshop.fake/cd#artist Data - Bonnie Tyler Identified subject - http://www.recshop.fake/cd/Hide your heart Metadata - http://www.recshop.fake/cd#year Data - 1988 Identified subject - http://www.recshop.fake/cd/Hide your heart Metadata - http://www.recshop.fake/cd#company Data - CBS Records Identified subject - http://www.recshop.fake/cd/Empire Burlesque Metadata - http://www.recshop.fake/cd#price Data - 10.90 Identified subject - http://www.recshop.fake/cd/Empire Burlesque Metadata - http://www.recshop.fake/cd#country Data - USA ***end code*** Open Source Tools for Chapter 6 ***begin code*** #!/usr/local/bin/perl use RDF::Notation3; use RDF::Notation3::Triples; $path = "image.n3"; $rdf = RDF::Notation3::Triples->new(); $rdf->parse_file($path); $triples = $rdf->get_triples; print @$triples->[0]->[0]; print "###\n"; use RDF::Notation3::XML; $rdf = RDF::Notation3::XML->new(); $rdf->parse_file($path); $string = $rdf->get_string; print $string; use RDF::Notation3; exit;[jb jb2_perl.txt] ***end code*** Open Source Tools for Chapter 6 ***begin code*** Hospital Unique_medical_identifier Numeric; Numeric -> Integer; Numeric -> Float; Integer -> Fixnum Integer -> Bignum } ***end code*** Open Source Tools for Chapter 6 ***begin code*** c:\ftp\dot>dot -Tpng number.dot -o number.png ***end code*** Open Source Tools for Chapter 6 ***begin code*** c:\ftp>twopi -Tpng neoplasms.dot -o neoplasms_classes.png ***end code*** Open Source Tools for Chapter 6 ***begin code*** #!/usr/bin/perl open (TEXT, "schema.txt"); open (OUT, ">schema.dot"); $/ = "\<\/rdfs\:Class>"; print OUT "digraph G \{\n"; print OUT "size\=\"15\,15\"\;\n"; print OUT "ranksep\=\"2\.00\"\;\n"; $line = " "; while ($line ne "") { $line = ; last if ($line !~ /\ $child\;\n"; print "$father \-\> $child\;\n"; } print OUT "\}"; exit; ***end code*** Open Source Tools for Chapter 6 ***begin code*** digraph G { size="15,15"; ranksep="2.00"; Class -> Tumor_classification; Tumor_classification -> Neoplasm; Tumor_classification -> Unclassified; Neural_tube -> Neural_tube_parenchyma; Mesoderm -> Sub_coelomic; Neoplasm -> Endoderm_or_ectoderm; Unclassified -> Syndrome; Neoplasm -> Neural_crest; Neoplasm -> Germ_cell; Neoplasm -> Pluripotent_non_germ_cell; Sub_coelomic -> Sub_coelomic_gonadal; Trophectoderm -> Molar; ***end code*** Section 7.3. ***begin code*** #!/usr/bin/ruby class Craniata def brag puts("I have a well-developed brain") end def myself puts("I am a member of Class " + self.class.to_s) end end class Gnathostomata < Craniata def speak puts("I have a jaw") end end class Teleostomi < Gnathostomata end class Mammalia < Teleostomi end class Theria < Mammalia end class Eutheria < Theria end class Canis < Eutheria def speak puts("Bow wow") end end class Primates < Eutheria def speak puts("Huf hufff") end end puts("Lassie") Lassie = Canis.new Lassie.speak Lassie.brag Lassie.myself puts() puts ("George_of_the_jungle") George_of_the_jungle = Primates.new George_of_the_jungle.speak George_of_the_jungle.brag George_of_the_jungle.myself puts() puts(Primates.method(:new).owner) puts(Canis.method(:new).owner) exit ***end code*** Section 7.3. ***begin code*** c:\ftp>lineage.rb Lassie Bow wow I have a well-developed brain I am a member of Class Canis George_of_the_jungle Huf hufff I have a well-developed brain I am a member of Class Primates Class Class ***end code*** Section 7.3. ***begin code*** Lassie = Canis.new George_of_the_jungle = Primates.new ***end code*** Section 7.3. ***begin code*** puts(Primates.method(:new).owner) puts(Canis.method(:new).owner) ***end code*** Section 7.3. ***begin code*** Class Class ***end code*** Section 7.3. ***begin code*** Lassie.speak Lassie.brag Lassie.myself ***end code*** Section 7.3. ***begin code*** def myself puts("I am a member of Class " + self.class.to_s) end ***end code*** Section 7.3. ***begin code*** class Gnathostomata < Craniata def speak puts("I have a jaw") end end class Canis < Eutheria def speak puts("I tell you that I am a member of Class Canis") end end class Primates < Eutheria def speak puts("I tell you that I am a member of Class Primates") end end ***end code*** Section 7.4. ***begin code*** 9f0ebdf2^^object_name^^Class 9f0ebdf2^^property^^subclass_of 9f0ebdf2^^property^^property 9f0ebdf2^^property^^definition 9f0ebdf2^^property^^object_name 9f0ebdf2^^property^^instance_of 9f0ebdf2^^subclass_of^^Class 9f0ebdf2^^instance_of^^Class 701cb7ed^^object_name^^Property 701cb7ed^^subclass_of^^Class 701cb7ed^^definition^^^^the metadata class 77cb79d5^^object_name^^instance_of 77cb79d5^^instance_of^^Property 77cb79d5^^definition^^the name of the class to which the object is an instance a03fbc3b^^object_name^^object_name a03fbc3b^^instance_of^^Property a03fbc3b^^definition^^word equivalent of its predicate identifying sequence de0e5aa1^^object_name^^subclass_of de0e5aa1^^instance_of^^Property de0e5aa1^^definition^^the name of the parent class of the referred object 4b675067^^object_name^^property 4b675067^^instance_of^^Property 4b675067^^definition^^an identifier a for class property c37529c5^^object_name^^definition c37529c5^^instance_of^^Property c37529c5^^definition^^the meaning of the referred object a29c59c0^^object_name^^dob a29c59c0^^instance_of^^Property a29c59c0^^definition^^date of birth, as Day, Month, Year a34a1e35^^object_name^^glucose_at_time a34a1e35^^instance_of^^Property a34a1e35^^definition^^glucose level in mg/Dl at time drawn (GMT) 03cc6948^^object_name^^Organism 03cc6948^^subclass_of^^Class 7d7ff42b^^object_name^^Hominidae 7d7ff42b^^subclass_of^^Organism 7d7ff42b^^property^^dob a0ce8ec6^^object_name^^Homo a0ce8ec6^^subclass_of^^Hominidae a0ce8ec6^^property^^glucose_at_time a1648579^^object_name^^Homo sapiens a1648579^^subclass_of^^Homo 98495efc^^object_name^^Andy Muzeack 98495efc^^instance_of^^Homo sapiens 98495efc^^dob^^1 January, 2001 98495efc^^glucose_at_time^^87, 02-12-2014 17:33:09 ***end code*** Section 7.4. ***begin code*** 7d7ff42b^^subclass_of^^Organism ***end code*** Section 7.4. ***begin code*** 7d7ff42b is the identifier subclass_of is the metadata Organism is the data ***end code*** Section 7.4. ***begin code*** 03cc6948^^object_name^^Organism 03cc6948^^subclass_of^^Class ***end code*** Section 7.4. ***begin code*** 98495efc^^object_name^^Andy Muzeack 98495efc^^instance_of^^Homo sapiens 98495efc^^dob^^1 January, 2001 98495efc^^glucose_at_time^^87, 02-12-2014 17:33:09 ***end code*** Section 7.4. ***begin code*** #!/usr/local/bin/perl open(TEXT, "triple.txt"); $line = " "; $object_name = "object_name"; $class_identifier = ""; $instance = "instance_of"; $property_class = "Property"; $property = "property"; while ($line ne "") { $line = ; $line =~ s/\n//o; @three = split(/\^\^/, $line) if ($line ne ""); $triple{$three[0]}{$three[1]}{$three[2]} = ""; } for $identifier (keys %triple) { if (exists($triple{$identifier}{$instance}{$property_class})) { @property_names = keys (%{$triple{$identifier}{$object_name}}); print "$property_names[0] is an instance of Class Property\n"; } } exit; ***end code*** Section 7.4. ***begin code*** subclass_of is an instance of Class Property instance_of is an instance of Class Property definition is an instance of Class Property object_name is an instance of Class Property glucose_at_time is an instance of Class Property property is an instance of Class Property dob is an instance of Class Property ***end code*** Section 7.4. ***begin code*** #/usr/local/bin/perl open(TEXT, "triple.txt"); $line = " "; $subclass = "subclass_of"; $object_name = "object_name"; $class_identifier = ""; $new_parent_identifier = $ARGV[0]; $class = $ARGV[0]; while ($line ne "") { $line = ; $line =~ s/\n//o; @three = split(/\^\^/, $line) if ($line ne ""); $triple{$three[0]}{$three[1]}{$three[2]} = ""; } for $identifier (keys %triple) { if (exists($triple{$identifier}{$object_name}{$class})) { $class_identifier = $identifier; last; } } @parent_array = keys (%{$triple{$class_identifier}{$subclass}}); print "$class is a subclass of $parent_array[0]"; exit; ***end code*** Section 7.4. ***begin code*** c:\ftp>parent.pl "Homo sapiens" Homo sapiens is a subclass of Homo c:\ftp>parent.pl "Homo" Homo is a subclass of Hominidae c:\ftp>parent.pl "Property" Property is a subclass of Class ***end code*** Open Source Tools for Chapter 7 ***begin code*** #!/usr/local/bin/perl use Data::Dump qw(dump); %hash = ( number => 42, string => 'This is a string', array => [ 1 .. 10 ], hash => { apple => 'red', banana => 'yellow'},); open(OUT, ">dump_struct"); print OUT dump \%hash; exit; ***end code*** Open Source Tools for Chapter 7 ***begin code*** #!/usr/local/bin/perl use Data::Dump qw(dump); open(IN, "dump_struct"); undef($/); $data = eval ; close $in; dump $data; exit; ***end code*** Open Source Tools for Chapter 7 ***begin code*** c:\ftp>data_slurp.pl { array => [1 .. 10], hash => { apple => "red", banana => "yellow" }, number => 42, string => "This is a string", } ***end code*** Open Source Tools for Chapter 7 ***begin code*** #!/usr/bin/python import pickle pumpkin_color = "orange" pickle.dump( pumpkin_color, open( "save.p", "wb" ) ) exit ***end code*** Open Source Tools for Chapter 7 ***begin code*** #!/usr/bin/python import pickle pumpkin_color = pickle.load( open( "save.p", "rb" ) ) print(pumpkin_color) exit ***end code*** Open Source Tools for Chapter 7 ***begin code*** c:\ftp\py>pickle_down.py orange ***end code*** Open Source Tools for Chapter 7 ***begin code*** #!/usr/bin/ruby class Shoestring < String def initialize @object_uuid = (`c\:\\cygwin64\\bin\\uuidgen.exe`).chomp end def object_uuid print @object_uuid end end loafer = Shoestring.new output = File.open("output_file.per", "wb") output.write(Marshal::dump(loafer)) exit ***end code*** Open Source Tools for Chapter 7 ***begin code*** #!/usr/bin/ruby class Shoestring < String def initialize @object_uuid = `c\:\\cygwin64\\bin\\uuidgen.exe`.chomp end def object_uuid print @object_uuid end end array = [] $/="\n\n" out = File.open("output_file.per", "rb").each do |object| array << Marshal::load(object) array.each do |object| puts object.object_uuid puts object.class puts object.class.superclass end end exit ***end code*** Open Source Tools for Chapter 7 ***begin code*** c:\ftp>object_demarshal.rb c2ace515-534f-411c-9d7c-5aef60f8c72a Shoestring String ***end code*** Open Source Tools for Chapter 7 ***begin code*** #!/usr/local/bin/perl use Fcntl; use SDBM_File; tie %lucy_hash, "SDBM_File", 'lucy', O_RDWR|O_CREAT|O_EXCL, 0644; $lucy_hash{"Fred Mertz"} = "Neighbor"; $lucy_hash{"Ethel Mertz"} = "Neighbor"; $lucy_hash{"Lucy Ricardo"} = "Star"; $lucy_hash{"Ricky Ricardo"} = "Band leader"; untie %lucy_hash; exit; ***end code*** Open Source Tools for Chapter 7 ***begin code*** #!/usr/local/bin/perl use Fcntl; use SDBM_File; tie %lucy_hash, "SDBM_File", 'lucy', O_RDWR, 0644; while(($key, $value) = each (%lucy_hash)) { print "$key => $value\n"; } untie %mesh_hash; exit; ***end code*** Open Source Tools for Chapter 7 ***begin code*** c:\ftp>lucy_untie.pl Fred Mertz => Neighbor Ethel Mertz => Neighbor Lucy Ricardo => Star Ricky Ricardo => Band leader ***end code*** Open Source Tools for Chapter 7 ***begin code*** #!/usr/local/bin/python import dumbdbm lucy_hash = dumbdbm.open('lucy', 'c') lucy_hash["Fred Mertz"] = "Neighbor" lucy_hash["Ethel Mertz"] = "Neighbor" lucy_hash["Lucy Ricardo"] = "Star" lucy_hash["Ricky Ricardo"] = "Band leader" lucy_hash.close() exit ***end code*** Open Source Tools for Chapter 7 ***begin code*** #!/usr/local/bin/python import dumbdbm lucy_hash = dumbdbm.open('lucy') for character in lucy_hash.keys(): print character, lucy_hash[character] lucy_hash.close() exit ***end code*** Open Source Tools for Chapter 7 ***begin code*** c:\ftp>lucy_untie.py Fred Mertz Neighbor Ethel Mertz Neighbor Lucy Ricardo Star Ricky Ricardo Band leader ***end code*** Open Source Tools for Chapter 7 ***begin code*** c:\>gem install gdbm ***end code*** Open Source Tools for Chapter 7 ***begin code*** #!/usr/local/bin/ruby require 'gdbm' lucy_hash = GDBM.new("lucy.db") lucy_hash["Fred Mertz"] = "Neighbor" lucy_hash["Ethel Mertz"] = "Neighbor" lucy_hash["Lucy Ricardo"] = "Star" lucy_hash["Ricky Ricardo"] = "Band leader" lucy_hash.close exit ***end code*** Open Source Tools for Chapter 7 ***begin code*** #!/usr/local/bin/ruby require 'gdbm' gdbm = GDBM.new("lucy.db") gdbm.each_pair do |name, role| print "#{name}: #{role}\n" end gdbm.close exit ***end code*** Open Source Tools for Chapter 7 ***begin code*** c:\ftp>lucy_untie.rb Ethel Mertz: Neighbor Lucy Ricardo: Star Ricky Ricardo: Band leader Fred Mertz: Neighbor ***end code*** Open Source Tools for Chapter 7 ***begin code*** #!/usr/local/bin/python import sqlite3 from sqlite3 import dbapi2 as sqlite import string, re, os mesh_hash = {} entry = () mesh_hash["Fred Mertz"] = "Neighbor" mesh_hash["Ethel Mertz"] = "Neighbor" mesh_hash["Lucy Ricardo"] = "Star" mesh_hash["Ricky Ricardo"] = "Band leader" con=sqlite.connect('test1.db') cur=con.cursor() cur.executescript(""" create table mesh ( name varchar(64), term varchar(64) ); """) for key, value in mesh_hash.iteritems(): entry = (key, value) cur.execute("insert into mesh (name, term) values (?, ?)", entry) con.commit() exit ***end code*** Open Source Tools for Chapter 7 ***begin code*** #!/usr/local/bin/python import sqlite3 from sqlite3 import dbapi2 as sqlite import string, re, os con=sqlite.connect('test1.db') cur=con.cursor() cur.execute("select * from mesh") for row in cur: print row[0], row[1] exit ***end code*** Open Source Tools for Chapter 7 ***begin code*** c:\ftp>sqlite6_read.py Fred Mertz Neighbor Ethel Mertz Neighbor Lucy Ricardo Star Ricky Ricardo Band leader ***end code*** Open Source Tools for Chapter 7 ***begin code*** #!/usr/local/bin/perl use DBI; $mesh_hash{"Fred Mertz"} = "Neighbor"; $mesh_hash{"Ethel Mertz"} = "Neighbor"; $mesh_hash{"Lucy Ricardo"} = "Star"; $mesh_hash{"Ricky Ricardo"} = "Band leader"; my $dbh = DBI->connect("dbi:SQLite:dbname=dbfile","",""); my $sth = $dbh->prepare("CREATE TABLE mesh (number VARCHAR(64), term VARCHAR(64))"); $sth->execute; $sth = $dbh->prepare("INSERT INTO mesh (number,term) VALUES(?,?)"); $dbh->do( "BEGIN TRANSACTION"); while ((my $key, my $value) = each(%mesh_hash)) { $sth->execute( $key, $value ); } $dbh->do( "COMMIT" ); exit; ***end code*** Open Source Tools for Chapter 7 ***begin code*** #!/usr/local/bin/perl use DBI; my $dbh = DBI->connect("dbi:SQLite:dbname=dbfile","",""); $sth = $dbh->prepare("SELECT number, term FROM mesh"); $sth->execute; while (@row = $sth->fetchrow_array()) { print "@row\n"; } exit; ***end code*** Open Source Tools for Chapter 7 ***begin code*** c:\ftp>perl_sqlite_out.pl Ricky Ricardo Band leader Lucy Ricardo Star Fred Mertz Neighbor Ethel Mertz Neighbor ***end code*** Open Source Tools for Chapter 7 ***begin code*** c:\>gem install sqlite3 ***end code*** Open Source Tools for Chapter 7 ***begin code*** #!/usr/local/bin/ruby require 'sqlite3' db = SQLite3::Database.new( "test.db" ) db_hash = Hash.new() db_hash["Fred Mertz"] = "Neighbor" db_hash["Ethel Mertz"] = "Neighbor" db_hash["Lucy Ricardo"] = "Star" db_hash["Ricky Ricardo"] = "Band leader" sql = <ruby_sqlite_out.rb Fred Mertz Neighbor Ethel Mertz Neighbor Lucy Ricardo Star Ricky Ricardo Band leader ***end code*** Section 8.1. ***begin code*** #!/usr/bin/perl $count = 0; while ($count < 600000) { $count++; $one_of_six = (int(rand(6))+1); $hash{$one_of_six}++; } while(($key, $value) = each (%hash)) { print "$key => $value\n"; } exit; ***end code*** Section 8.1. ***begin code*** $one_of_six = (int(rand(6))+1); ***end code*** Section 8.1. ***begin code*** C:\ftp>perl randtest.pl 1 => 100002 2 => 99902 3 => 99997 4 => 100103 5 => 99926 6 => 100070 ***end code*** Section 8.1. ***begin code*** C:\ftp>perl randtest.pl 1 => 100766 2 => 99515 3 => 100157 4 => 99570 5 => 100092 6 => 99900 ***end code*** Section 8.1. ***begin code*** #!/usr/bin/perl for ($count = 1; $count <= 12; $count++) { push(@listchar, chr(int(rand(26))+65)); } $listchar[8]= "."; $randomfilename = join("",@listchar); print "Your filename is $randomfilename\n"; exit; ***end code*** Section 8.1. ***begin code*** c:\ftp>random_filenames.pl Your filename is OAOKSXAH.SIT ***end code*** Section 8.1. ***begin code*** push(@listchar, chr(int(rand(26))+65)); ***end code*** Section 8.1. ***begin code*** #!/usr/bin/python import random filename = [0]*12 filename = map(lambda x: x is "" or chr(int(random.uniform(0,25) + 65)), filenam e) print ''.join(filename[0:8]) + "." + ''.join(filename[9:12]) exit ***end code*** Section 8.1. ***begin code*** c:\ftp>random_filenames.py KYSDWKLF.RBA ***end code*** Section 8.1. ***begin code*** #!/usr/bin/python import random from math import sqrt totr = 0 totsq = 0 for iterations in range(10000000): x= random.uniform(0,1) y= random.uniform(0,1) r= sqrt((x*x) + (y*y)) if r < 1: totr = totr + 1 totsq = totsq + 1 print float(totr)*4.0/float(totsq) exit ***end code*** Section 8.1. ***begin code*** output: c:\ftp\py>pi.py 3.1414256 ***end code*** Section 8.1. ***begin code*** #!/usr/local/bin/ruby x = y = totr = totsq = 0.0 (1..100000).each do x = rand() y = rand() r = Math.sqrt((x*x) + (y*y)) totr = totr + 1 if r < 1 totsq = totsq + 1 end puts (totr *4.0 / totsq) exit ***end code*** Section 8.1. ***begin code*** #!/usr/local/bin/perl open(DATA,">pi.dat"); for (1..10000) { $x = rand(); $y = rand(); $r = sqrt(($x*$x) + ($y*$y)); if ($r < 1) { $totr = $totr + 1; print DATA "$x\ $y\n"; } $totsq = $totsq + 1; } print eval(4 * ($totr / $totsq)); exit; ***end code*** Section 8.1. ***begin code*** #!/usr/local/bin/perl open (OUT, ">x_dist.dat"); for($i=0;$i<1000;$i++) { $column[$i][0] = $i; $column[$i][1] = 0; } for($i=0;$i<1000;$i++) { $n = 0; $x = 0; while($n < 1000) { $x = $x + rand(1); $n = $n + 1; } $x = int($x); $column[$x][1] = $column[$x][1] + 1; } for($i=0;$i<1000;$i++) { print OUT "$column[$i][0] $column[$i][1]\n"; } close OUT; $command = "gnuplot.exe \-e \"set term png; set output \'c\:\\ftp\\out.png\'; plot \'c\:\\ftp\\x_dist.dat\' smooth bezier\""; chdir "c\:\\Program Files \(x86\)\\gnuplot\\bin\\"; system($command); exit; ***end code*** Section 8.2. ***begin code*** #!/usr/local/bin/perl $death_chance = 0.49; @cell_array = [1]; @cell_array_incremented = [1,1,1]; while(length(@cell_array)>0) { foreach $cell (@cell_array) { $randnum = int( rand(100) ) + 1; if ($randnum > (100 * $death_chance)) { push(@cell_array_incremented, 1); } else { shift(@cell_array_incremented); } } die "no more cells in colony" if (scalar(@cell_array_incremented) == 0); $cell_array = @cell_array_incremented; print scalar(@cell_array_incremented) . "\,"; } exit; ***end code*** Section 8.2. ***begin code*** if ($randnum > (100 * $death_chance)) { push(@cell_array_incremented, 1); } else { shift(@cell_array_incremented); } ***end code*** Section 8.2. ***begin code*** c:\ftp>clone.pl 2,3,4,5,4,5,4,5,6,7,8,9,8,9,10,9,8,9,8,9,8,9,8,7,6,5,6,7,6,5,6,7,8,9,8,9,8,9,8,9 ,8,9,10,11,10,11,12,11,10,11,10,11,10,9,10,9,8,7,8,7,8,9,10,11,10,11,12,13,12,11 ,10,9,10,11,10,11,12,11,12,11,10,11,12,13,12,11,10,9,8,9,10,9,10,11,12,11,12,13, 14,15,16,15,14,13,12,11,10,9,8,7,6,7,8,9,8,7,6,5,4,3,4,5,6,7,8,9,10,9,10,9,10,11 ,12,13,14,13,12,11,10,9,10,11,10,11,12,13,14,13,12,13,12,11,12,13,12,13,14,15,16 ,15,16,15,14,15,14,13,12,13,14,13,14,13,14,15,14,15,14,13,14,15,16,15,14,13,14,1 3,12,13,14,13,14,13,14,15,16,15,14,13,12,11,10,11,10,11,10,9,8,9,8,7,6,7,8,9,8,9 ,10,11,12,11,12,11,10,11,12,13,14,15,16,15,14,13,12,11,10,11,10,9,10,9,8,7,6,7,8 ,7,8,7,6,5,6,5,6,7,6,7,8,7,6,7,8,7,8,7,6,5,4,3,2,3,2,3,2,3,4,5,4,5,4,3,4,3,4,3,2 ,1, ***end code*** Section 8.2. ***begin code*** c:\ftp>clone.pl 2,1,2,1,2,3,4,5,4,3,2,3,4,5,6,7,8,9,10,11,10,11,12,13,14,13,14,13,14,13,14,15,14 ,15,16,15,14,15,14,15,14,15,16,15,14,15,16,15,14,15,14,15,14,15,16,17,18,19,18,1 7,18,17,18,19,18,19,18,17,18,19,18,19,20,19,20,21,20,19,20,21,20,19,18,19,20,19, 18,19,20,19,20,19,18,17,18,17,16,17,16,15,14,15,14,13,12,11,10,11,10,9,10,11,12, 11,12,11,12,13,14,15,14,13,14,15,16,15,14,15,16,17,18,17,16,15,14,13,14,15,16,15 ,16,17,16,17,16,17,18,17,16,15,14,13,12,11,10,11,10,11,12,11,10,11,12,11,10,11,1 2,11,10,9,8,9,10,9,8,7,6,5,6,5,6,7,8,9,8,7,6,7,8,7,8,7,6,5,6,5,6,7,6,7,6,7,8,9,8 ,9,10,11,12,13,12,11,10,9,10,11,10,9,8,7,8,9,10,11,12,11,12,13,12,13,12,11,12,11 ,12,13,12,11,10,11,10,9,8,7,8,9,10,9,10,9,8,7,6,5,6,7,6,7,6,7,8,7,8,7,8,7,6,7,8, 9,8,7,8,9,10,9,10,11,12,11,12,13,14,13,12,13,14,13,12,13,12,13,12,11,10,11,10,11 ,10,11,12,11,10,11,10,9,10,9,8,9,10,11,10,9,8,9,8,7,8,9,8,7,8,7,8,9,10,11,10,11, 12,11,10,11,12,11,10,11,12,11,10,11,12,11,12,11,10,9,8,9,8,7,6,7,6,7,8,7,6,5,4,5 ,4,3,2,1, ***end code*** Section 8.2. ***begin code*** #!/usr/local/bin/python import numpy.random as npr import sys death_chance = 0.51; cell_array = [1, 1, 1] cell_array_incremented = [1,1,1] while(len(cell_array) > 0): for cell in cell_array: randnum = npr.randint(1,101) if randnum > 100 * death_chance: cell_array_incremented.append(1) else: cell_array_incremented.remove(1) if len(cell_array_incremented) < 1: sys.exit() if len(cell_array_incremented) > 400: sys.exit() cell_array = cell_array_incremented print len(cell_array_incremented), exit ***end code*** Section 8.2. ***begin code*** #!/usr/local/bin/ruby death_chance = 0.51 cell_array = [1, 1, 1] cell_array_incremented = [1,1,1] while(cell_array.length > 0) cell_array.each do randnum = rand(101).to_i if randnum > 100 * death_chance cell_array_incremented.push(1) else cell_array_incremented.pop(1) end end if cell_array_incremented.length < 1 abort end if cell_array_incremented.length > 400 abort end cell_array = cell_array_incremented print cell_array_incremented.length print " " end exit ***end code*** Section 8.2. ***begin code*** #!/usr/local/bin/perl for($trial=0;$trial<10000;$trial++) { $group_likelihood = rand(); if ($group_likelihood < 0.2) #puts trial in poor-judgment group { $bad_likelihood = rand(); #let's roll the dice to see if an accident occurs if ($bad_likelihood < 0.6) #an accident occurred, simulating an initial condition of poor-judgment with accident { $next_bad_likelihood = rand(); if ($next_bad_likelihood < 0.6) { $accidents_next_year = $accidents_next_year + 1; #let's add 1 to the tally of trials with accidents } else { $no_accidents_next_year = $no_accidents_next_year + 1; #let's add 1 to the tally of trials with no accidents } } } else #if $group_likelihood is greater than 0.2, we bump the trial into the good-judgment group { $bad_likelihood = rand(); #let's roll the dice to see if an accident occurs if ($bad_likelihood < 0.06) #an accident occurred, simulating an initial condition of good-judgment with accident { $next_bad_likelihood = rand(); if ($next_bad_likelihood < 0.06) { $accidents_next_year = $accidents_next_year + 1; #let's add 1 to the tally of trials with accidents } else { $no_accidents_next_year = $no_accidents_next_year + 1; #let's add 1 to the tally of trials with no accidents } } } } $cost = int((($accidents_next_year) / ($accidents_next_year + $no_accidents_next_year) * 100 * 100)); print "Insurance cost is $cost"; exit; ***end code*** Section 8.2. ***begin code*** c:\ftp>bayes_accident.pl Insurance cost is $4527 c:\ftp>bayes_accident.pl Insurance cost is $4301 c:\ftp>bayes_accident.pl Insurance cost is $4514 c:\ftp>bayes_accident.pl Insurance cost is $4669 c:\ftp>bayes_accident.pl Insurance cost is $4360 c:\ftp>bayes_accident.pl Insurance cost is $4550 c:\ftp>bayes_accident.pl Insurance cost is $4340 c:\ftp>bayes_accident.pl Insurance cost is $4480 ***end code*** Section 8.2. ***begin code*** #!/usr/local/bin/perl for($trial=0;$trial<10000;$trial++) { undef(%date_hash); for($i=0;$i<30;$i++) { $date = int(rand(365)); $date_hash{$date} = $date_hash{$date} + 1; if ($date_hash{$date} == 2) { $success = $success + 1; last; } } } print ($success / $trial); exit; ***end code*** Section 8.2. ***begin code*** c:\ftp>rand_birthday.pl 0.7076 ***end code*** Section 8.2. ***begin code*** #!/usr/local/bin/ruby success = date = 0 date_hash = Hash.new (1..10000).each do date_hash.clear (1..30).each do |date| date = rand(365) if (date_hash.has_key?(date)) success = success + 1 break else date_hash[date] = 1 end end end puts success.fdiv(10000) exit ***end code*** Section 8.2. ***begin code*** c:\ftp>rand_birthday.rb 0.7004 ***end code*** Section 8.3. ***begin code*** #!/usr/local/bin/python import numpy as np sample = np.arange(100) gather = [] for i in sample: np.random.shuffle(sample) print sample exit ***end code*** Section 8.3. ***begin code*** c:\ftp>shuffle_100.py [70 7 74 96 39 27 1 86 55 79 80 32 22 95 50 59 26 5 92 64 71 6 91 0 73 56 13 99 87 20 2 81 14 36 29 67 46 12 84 15 57 24 98 62 52 35 34 61 8 77 37 43 30 40 76 48 68 90 41 33 88 58 25 17 82 83 4 54 47 31 97 10 85 75 45 72 19 42 65 9 28 51 16 89 44 3 53 49 69 63 38 23 93 11 94 60 78 66 21 18] ***end code*** Section 8.3. ***begin code*** #!/usr/bin/perl use List::Util qw(shuffle); @data = (1..10); print join(",", shuffle(@data)); exit; ***end code*** Section 8.3. ***begin code*** c:\ftp>shuffle.pl 5,6,2,9,4,3,8,10,1,7 c:\ftp>shuffle.pl 10,8,3,1,7,6,2,9,4,5 c:\ftp>shuffle.pl 8,10,3,1,9,4,5,6,2,7 c:\ftp>shuffle.pl 1,3,5,9,10,6,7,4,8,2 ***end code*** Section 8.3. ***begin code*** #!/usr/bin/ruby print (1..10).to_a.shuffle! exit ***end code*** Section 8.3. ***begin code*** c:\ftp>shuffle.rb [1, 9, 7, 4, 5, 3, 2, 10, 6, 8] c:\ftp>shuffle.rb [4, 6, 9, 1, 7, 3, 8, 5, 10, 2] c:\ftp>shuffle.rb [3, 7, 2, 8, 1, 9, 10, 6, 5, 4] c:\ftp>shuffle.rb [5, 9, 7, 3, 6, 10, 8, 1, 4, 2] c:\ftp>shuffle.rb [1, 7, 9, 3, 8, 5, 6, 10, 2, 4] ***end code*** Section 8.3. ***begin code*** #!/usr/local/bin/python import numpy as np from random import randint low_array = [] high_array = [] for i in range(50): low_array.append(randint(1,80)) print "Here's the low-skewed data set " + str(low_array) for i in range(50): high_array.append(randint(21,100)) print "\nHere's the high-skewed data set " + str(high_array) av_diff = (sum(high_array)/len(high_array)) - (sum(low_array)/len(low_array)) print "\nHere's the difference in the average value of the two arrays " + str(av_diff) exit ***end code*** Section 8.3. ***begin code*** c:\ftp>low_high.py Here's the low-skewed data set [75, 16, 32, 60, 59, 33, 4, 63, 50, 26, 59, 70, 6 6, 26, 31, 64, 24, 58, 15, 5, 37, 68, 10, 29, 16, 31, 21, 38, 67, 8, 15, 70, 1, 42, 75, 53, 59, 75, 13, 55, 25, 80, 53, 46, 65, 70, 30, 25, 6, 18] Here's the high-skewed data set [76, 36, 23, 35, 44, 91, 78, 82, 22, 88, 90, 34, 78, 30, 65, 96, 77, 56, 44, 23, 48, 70, 96, 46, 30, 59, 56, 31, 40, 89, 70, 64, 44, 83, 51, 53, 32, 98, 30, 74, 33, 63, 23, 31, 24, 85, 34, 68, 53, 65] Here's the difference in the average value of the two arrays 16 ***end code*** Section 8.3. ***begin code*** #!/usr/local/bin/python import numpy as np from random import randint low_array = [] high_array = [] gather = [] for i in range(50): low_array.append(randint(1,80)) for i in range(50): high_array.append(randint(21,100)) av_diff = (sum(high_array)/len(high_array)) - (sum(low_array)/len(low_array)) print "The difference in the average value of the high and low arrays is: " + str(av_diff) sample = low_array + high_array for i in sample: np.random.shuffle(sample) right = sample[50:] left = sample[:50] gather.append(abs((sum(left)/len(left)) - (sum(right)/len(right)))) sorted_gather = sorted(gather) print "The 5 smallest and 5 largest differences of averages of left and right arrays are:" print str(sorted_gather[:5]) + " and " + str(sorted_gather[95:]) exit ***end code*** Section 8.3. ***begin code*** c:\ftp>pop_diff.py The difference in the average value of the high and low arrays is: 18 The 5 smallest and 5 largest differences of averages of left and right arrays are: [0, 0, 0, 0, 0] and [9, 10, 11, 11, 12] ***end code*** Section 8.3. ***begin code*** c:\ftp>low_high.py Here's the difference in the average value of the two arrays 20 c:\ftp>low_high.py Here's the difference in the average value of the two arrays 23 c:\ftp>low_high.py Here's the difference in the average value of the two arrays 19 c:\ftp>low_high.py Here's the difference in the average value of the two arrays 12 c:\ftp>low_high.py Here's the difference in the average value of the two arrays 22 c:\ftp>low_high.py Here's the difference in the average value of the two arrays 21 c:\ftp>low_high.py Here's the difference in the average value of the two arrays 20 c:\ftp>low_high.py Here's the difference in the average value of the two arrays 15 ***end code*** Section 8.3. ***begin code*** #!/usr/local/bin/python import numpy as np import numpy.random as npr first_array = [] total_array = [] total = 0 for i in range(50): first_array.append(npr.randint(1,100)) data = np.array(first_array) for i in range(50): idx = npr.randint(0, len(data), (1,len(data))) samples_with_replacement = np.array(data[idx]).ravel() print (sum(samples_with_replacement)/len(samples_with_replacement)), total = total + (sum(samples_with_replacement)/len(samples_with_replacement)) print "\n" print total/50 exit ***end code*** Section 8.3. ***begin code*** c:\ftp>resample.py 54 54 59 56 47 48 55 57 52 50 43 53 46 55 55 48 49 49 56 45 55 59 51 45 43 49 51 54 48 54 56 45 52 54 57 57 51 51 44 51 45 49 54 50 52 52 55 61 51 54 51 c:\ftp>resample.py 47 58 59 44 53 43 58 43 47 48 45 48 49 45 56 51 52 52 55 53 47 48 43 53 54 49 50 61 52 59 49 58 51 56 53 53 51 56 54 53 49 44 48 51 49 45 44 50 52 55 50 c:\ftp>resample.py 55 49 57 52 51 48 49 58 47 52 47 47 45 51 45 52 54 53 49 47 50 52 46 51 48 46 49 54 53 53 53 51 43 54 51 54 57 52 46 49 53 55 52 51 56 57 47 53 47 45 50 ***end code*** Open Source Tools for Chapter 8 ***begin code*** #!/usr/bin/python input_string = "four score and seven years ago our fathers brought forth upon" input_string = input_string + " this continent a new nation conceived in liberty and" input_string = input_string + "\0" table = sorted(input_string[i:] + input_string[:i] for i in range(len(input_string))) last_column = [row[-1:] for row in table] print("".join(last_column)) exit ***end code*** Open Source Tools for Chapter 8 ***begin code*** c:\ftp>bwt.py dtsyesnsrtdnwaordnhn efni n snenryvcvnhbsn uatttgl tthe oioe oaai eogipccc fr fuuuobaeoerri nhra naro ooieet ***end code*** Open Source Tools for Chapter 8 ***begin code*** four score and seven years ago our fathers brought forth upon this continent a new nation conceived in liberty and dtsyesnsrtdnwaordnhn efni n snenryvcvnhbsn uatttgl tthe oioe oaai eogipcccfr fuuuobaeoerri nhra naro ooieet ***end code*** Open Source Tools for Chapter 8 ***begin code*** #!/usr/bin/python input_string = "four score and seven years ago our fathers brought forth upon" input_string = input_string + " this continent a new nation conceived in liberty and" input_string = input_string + "\0" table = sorted(input_string[i:] + input_string[:i] for i in range(len(input_string))) last_column = [row[-1:] for row in table] #The first lines re-created the forward BWT #The next four lines compute the inverse transform table = [""] * len(last_column) for i in range(len(last_column)): table = sorted(last_column[i] + table[i] for i in range(len(input_string))) print([row for row in table if row.endswith("\0")][0]) exit ***end code*** Open Source Tools for Chapter 8 ***begin code*** c:\ftp>bwt_inverse.py four score and seven years ago our fathers brought forth upon this continent a new nation conceived in liberty and ***end code*** Open Source Tools for Chapter 8 ***begin code*** #!/usr/bin/python import numpy as np input_string = "\0 four score and seven years ago our fathers brought forth upon" input_string = input_string + " this continent a new nation conceived in liberty and" word_list = input_string.rsplit() table = sorted(word_list[i:] + word_list[:i] for i in range(len(word_list))) last_column = [row[-1:] for row in table] first_column = [row[:1] for row in table] print "This is the first column of the transform table:\n" + str(first_column) + "\n" #The first lines of the script create the forward BWT #The next four lines compute the inverse transform table = [""] * len(last_column) for i in range(len(last_column)): table = sorted(str(last_column[i]) + " " + str(table[i]) for i in range(len(word_list))) original_string = [row for row in table][0] print "Here is the inverse transform, as a word array:\n" + str(original_string) exit ***end code*** Open Source Tools for Chapter 8 ***begin code*** c:\ftp>bwt_trans_inv.py This is the first column of the transform table: [['\x00'], ['a'], ['ago'], ['and'], ['and'], ['brought'], ['conceived'], ['continent'], ['fathers'], ['forth'], ['four'], ['in'], ['liberty'], ['nation'], ['new'], ['our'], ['score'], ['seven'], ['this'], ['upon'], ['years']] Here is the inverse transform, as a word array: ['\x00'] ['four'] ['score'] ['and'] ['seven'] ['years'] ['ago'] ['our'] ['fathers'] ['brought'] ['forth'] ['upon'] ['this'] ['continent'] ['a'] ['new'] ['nation'] ['conceived'] ['in'] ['liberty'] ['and'] ***end code*** Glossary ***begin code*** !"#$%&'()*+,-./0123456789:;<=> ?@ABCDEFGHIJKLMNOPQRSTUVWXYZ [\]^_`abcdefghijklmnopqrstuvwxyz{|}~ ***end code*** Glossary ***begin code*** #!/usr/local/bin/ruby my_hash = Hash.new my_hash["C05"] = "Albumin" my_hash["C39"] = "Choline" my_hash.each {|key,value| STDOUT.print(key, " --- ", value, "\n")} exit ***end code*** Glossary ***begin code*** Output: C05 --- Albumin C39 --- Choline ***end code*** Glossary ***begin code*** #!/usr/bin/python import base64 import md5 md5_object = md5.new() sample_file = open ("z.txt", "rb") string = sample_file.read() sample_file.close() md5_object.update(string) md5_string = md5_object.digest() print(base64.encodestring(md5_string)) exit ***end code*** Glossary ***begin code*** 150 ***end code*** Glossary ***begin code*** for($i=0;$i<1000;$i++) ***end code*** Glossary ***begin code*** class Salutations def hello puts "hello there" end end ***end code*** Glossary ***begin code*** #!/usr/local/bin/ruby require 'digest/md5' file_contents = File.new("simplify.txt").binmode hash_string = Digest::MD5.base64digest(file_contents.read) puts hash_string exit ***end code*** Glossary ***begin code*** 0CfZez7L1A6WFcT+oxMh+g== ***end code*** Glossary ***begin code*** #!/usr/local/bin/perl #!/usr/local/bin/ruby #!/usr/local/bin/python #!/usr/local/bin/sh ***end code*** Glossary ***begin code*** urn:isbn-13:9780128028827 ***end code*** Glossary ***begin code*** urn:uuid:e29d0078-f7f6-11e4-8ef1-e808e19e18e5 ***end code***