10 Linux Performance Monitoring and Debugging Tools

Similar to the previous post, this short post provides 10 monitoring and debugging tools that will be helpful when you are working on a Linux environment.

Sar	#Report I/O statistics every 1 second for a total of 3 times sar –b 1 3
Tcpdump	# Displays captured packages in ASCII tcpdump –A –i eth0
Iostat	#Displays information about the CPU usage and I/O statistics about all the partitions of the system iostat
Mpstat	#Report processor statistics mpstat -A
Vmstat	#Display the memory usage (including swap) vmstat
PS	#Display processes that belong to the specified usernames ps –f –u wwwrun, postfix #Display the process id and commands in a hierarchy. ps –e –o pid,args --forest
Free	#Display the total physical memory on this system in KB free #Display total in MB displaying the sum of physical and swap memory values, excluding the buffer/cache line. free –mto
TOP	# Displays a specific user process only top –u geek While unix top command is running, press u which will ask for username.
Pmap	#Displays memory of the given process id pmap 5732 #Display additional information about the memory maps pmap –x 5732 #Display the device information of the process maps pmap –d pid

10 Frequently Used UNIX / Linux Commands

This short post provides practical examples for ten most frequently used commands in Linux/UNIX. The purpose of this list is to give you a jump-start on some of the common Linux commands.

Tar	#Create a new tar archive tar cvf archive_name.tar dirname/ #Extract from an existing tar archive tar xvf archive_name.tar #View an existing tar archive tar tvf archive_name.tar
Grep	#Search for a given string in a file (case insensitive) grep –i “the” demo_file #Print the matched line along with the 3 lines after it grep –A 3 –i “example” demo_text #Search for a given string in all files recursively grep –r “ramesh” *
Find	#Find files using file-name (case insensitive) find –iname “MyProgram.c” #Execute commands on the files found by the find command find –iname “MyProgram.c” –exec md5sum {} \; #Find all empty files in home directory find ~ -empty
SSH	#Login to remote host ssh –l jsmith remotehost.example.com #Debug ssh client ssh –v –l jsmith remotehost.example.com #Display SSH client version ssh –V
Sed	#Convert the DOS file format to Unix file format sed ‘s/.$//’filename #Print file content in reverse sed –n ‘1!G;h;$p’ thegeekstuff.txt #Add line number for all non-empty-lines in a file sed ‘/./=’ thegeekstuff.txt | sed ‘N; s/\n/ /’
Awk	#Remove duplicate lines using awk awk ‘!($0 in array) { array[$0]; print }’ temp # Print all lines from /etc/passwd that have the same uid and gid awk –F ‘:’ ‘$3 == $4’ passwd.txt # Print only a specific field from a file awk ‘{print $2, $5}’ employee.txt
Vim	#Go to the 143rd line of file vim +143 filename.txt #Go to the first match of the specified vim +/search-term filename.txt #Open file in read-only mode vim –R /etc/passwd
Diff	#Ignore white space while comparing diff –w name_list.txt name_list new.txt
Sort	#Sort a file in ascending order sort names.txt #Sort a file in descending order sort –r names.txt #Sort passwd file by 3rd field sort –t: -k 3n /etc/passwd | more
Export	#View oracle related environment variables export | grep ORACLE #Export an environment variable export ORACLE_HOME = ‘/u01/app/oracle/product/10.2.0’

Computer Architectures

One way of categorizing computer architectures is by number of instructions executed per clock. Many computing machines read one instruction at a time and execute it (or they put a lot of effort into acting as if they do that, even if internally they do fancy superscalar and out-of-order stuff). I call such machines "von Neumann" machines, because all of them have a von Neumann bottleneck. Such machines include CISC, RISC, MISC, TTA, and DSP architectures. Such machines include accumulator machines, register machines, and stack machines. Other machines read and execute several instructions at a time (VLIW, super-scalar), which break the one-instruction-per-clock limit, but still hit the von Neumann bottleneck at some slightly larger number of instructions-per-clock. Yet other machines are not limited by the von Neumann bottleneck, because they pre-load all their operations once at power-up and then process data with no further instructions. Such non-Von-Neumann machines include dataflow architectures, such as systolic architectures and cellular automata, often implemented with FPGAs, and the NON-VON supercomputer.

Another way of categorizing computer architectures is by the connection(s) between the CPU and memory. Some machines have a unified memory, such that a single address corresponds to a single place in memory, and when that memory is RAM, one can use that address to read and write data, or load that address into the program counter to execute code. I call these machines Princeton machines. Other machines have several separate memory spaces, such that the program counter always refers to "program memory" no matter what address is loaded into it, and normal reads and writes always go to "data memory", which is a separate location usually containing different information even when the bits of the data address happen to be identical to the bits of the program memory address. Those machines are "pure Harvard" or "modified Harvard" machines. Most DSPs have 3 separate memory areas -- the X ram, the Y ram, and the program memory. The DSP, Princeton, and 2-memory Harvard machines are three different kinds of von Neumann machines. A few machines take advantage of the extremely wide connection between memory and computation that is possible when they are both on the same chip -- computational ram or iRAM or CAM RAM -- which can be seen as a kind of non-von Neumann machine.

A few people use a narrow definition of "von Neumann machine" that does not include Harvard machines. If you are one of those people, then what term would you use for the more general concept of "a machine that has a von Neumann bottleneck", which includes both Harvard and Princeton machines, and excludes NON-VON?

Most embedded systems use Harvard architecture. A few CPUs are "pure Harvard", which is perhaps the simplest arrangement to build in hardware: the address bus to the read-only program memory is exclusively is connected to the program counter, such as many early Microchip PICmicros. Some modified Harvard machines, in addition, also put constants in program memory, which can be read with a special "read constant data from program memory" instruction (different from the "read from data memory" instruction). The software running in the above kinds of Harvard machines cannot change the program memory, which is effectively ROM to that software. Some embedded systems are "self-programmable", typically with program memory in flash memory and a special "erase block of flash memory" instruction and a special "write block of flash memory" instruction (different from the normal "write to data memory" instruction), in addition to the "read data from program memory" instruction. Several more recent Microchip PICmicros and Atmel AVRs are self-programmable modified Harvard machines.

Another way to categorize CPUs is by their clock. Most computers are synchronous -- they have a single global clock. A few CPUs are asynchronous -- they don't have a clock -- including the ILLIAC I and ILLIAC II, which at one time were the fastest supercomputers on earth.

Practical cryptography in 2011

Before anyone objects, I'm going to clarify the ground rules. This list is naturally incomplete, and moreover, I'm going to take 'practical' seriously. That rules out reduced-round attacks (on anything); improvements in Fully-Homomorphic Encryption (getting faster!); and any paper with 'composable' in the title. I will cover implementation and usability issues. But I don't really plan to take any of my own rules that seriously. If you think I missed something important -- and I will -- please feel free to follow up in comments.

·     SecurID not so secure after all. RSA's SecurID has become practically synonymous with two-factor authentication. And it's not a bad system. Back in 2010 if you'd asked me about the most likely avenue for a compromise, I would have guessed (a) theft of secrets from a local SecurID server, or (b) some kind of bug in the authentication software.

What I wouldn't have guessed was (c) compromise of RSA's master seed data. Obviously that wasn't going to happen -- given the sensitivity of this information, RSA naturally took good care of it. Remember the underground research facility in Resident Evil? My vision of the RSA master seed storage facility was kind of like that, only without the friendly computer or Milla Jovovich.

In retrospect this seems a bit naive, but really: SecurID was everywhere. Even military contractors used it. If local banks had learned to store their secrets in hardware security modules, then certainly RSA would take modest precautions with something as important as their master seed database.

What 2011 taught us is that just because you think something will be done a certain way, it won't necessarily be so. Perhaps 2012 will be different.
 

·         Still more attacks on AES. I promised to keep this to the practical, and the latest attacks against AES certainly aren't. The best of them still only shave a couple of bits off of the security of an AES key (although the new attacks don't depend on related keys).

Still, AES is such an important cipher that any reduction in its security margin is going to have some kind of practical impact. The question now is what lies down the road: (a) an improved AES beefed up with additional rounds, (b) a new standards competition, (c) apathy, or (d) something even weirder. Maybe 2012 will put us on a path to finding that answer.
 

·         D-Wave actually builds a quantum computer. With two caveats: it's not really a computer, and it may or may not be quantum.

In case you've never heard of D-Wave, they're a private company that purports to sell the first working adiabatic quantum computer, complete with "128 pair-wise coupled superconducting flux qubits". If you're not sure what to make of this, you're not the only one. Thanks to D-Wave's NDA-heavy business model, there's been some doubt in learned quarters as to whether the D-Wave device actually performs useful quantumcomputation at all.

But D-Wave has an active and respectable research department. Just this past May they published an article in Nature demonstrating apparent quantum tunnelingamongst eight qubits. This is a far cry from the 128 qubits claimed in their commercial products, it doesn't demonstrate entanglement, and of course, it doesn't result in computation. Still it's not nothing. So does this mean practical implementations ofShor's algorithm are just around corner? Probably not.
  

·      TLS falls to the BEAST. This fall Thai Duong and Juliano Rizzo raised the pulse of the security industry by demonstrating the most exciting practical attack againstSSL/TLS in years. This attack is dear to my heart because it's one of the first attacks I wrote about on this blog.

BEAST is based on a five year-old academic attack on a twelve year old standard. Now there's a bit more to the attack that Rizzo/Duong implemented, but still --- how in the world does such a thing stay in the wild so long? A colleague describes Rizzo/Duong's research strategy as 'working their way through the Eurocrypt archives, actually exploiting the theoretical vulnerabilities that academics couldn't be bothered to follow through on'. If this sounds dismissive, it wasn't -- he only wished that he'd done it first.

Lesson: there's a hell of a gap between academic crypto and 'the real world'. We're only safe as long as you believe in attackers who are sophisticated enough to spear-phish SecurID, but not bright enough to read a crypto paper. Time will tell if that belief is a good one.
 

·     Nothing much happens with hash functions. Well, that's a bit unfair. Thingsdid happen, and there are some papers to show for it. The SHA3 finalists, announced late last year, continued to percolate through the system. (Go BLAKE!) Still, 2011 was a huge disappointment for those of us you who thought that it would be the year of SHA1 collisions. I guess there's always 2012.
 

·      Unsigned code running on iOS. Many cryptographers would exclude this from a list of 'crypto' happenings. Still, a theme of this blog is that your crypto doesn't matter if your implementation undermines it.

This is exactly what Charlie Miller demonstrated when he showed how to execute malicious application code on an iPhone. Even though iOS devices are only supposed to run signed code, it turns out that a few bad checks created a tiny section of memory in which these requirements were irrelevant. To thank him, Apple created atiny section of their developer program in which Charlie Miller is irrelevant.
 

·        Certificate Authorities still a giant mess. Some will say I'm exaggerating -- there were only a few major hacks this year, notably Dutch CA DigiNotar, and a smaller CA called Gemnet. But don't forget, Microsoft also revoked Digicert Malaysia -- not because they were hacked, but for using 512-bit RSA keys to sign certificates (!!) And this is only a partial list of the problems we know about. All in all, it was not a good year to be a CA.
 

·      Secure police radios not all that secure. APCO Project 25 is a digital radio standard used by law enforcement. Naturally it supports encryption -- a valuable tool when you're discussing a confidential source or razzing those Occupy protestors. And the P25 encryption itself isn't half bad (it's unauthenticated AES). Rather, the problem with P25 is that simply getting encryption turned on is a daunting prospect -- to the point where it often doesn't happen.
 
To illustrate the problem, a team from UPenn spent two years snarfing up confidential traffic. The agents generating most of this traffic either thought it was encrypted, or would have encrypted it if only they could've figured out how to set the keys. Still more evidence that usability is every bit as important as crypto, implementation, or any other issue that we come across in our field.
 

·      XML Encryption nearly as awful as XML itself. This fall brought news of a clever attack on the unauthenticated W3C XML Encryption standard. The attack, by researchers Tibor Jager and Juraj Somorovsky, used various features of XML encoding to implement what I would call a 'padding oracle attack-plus-plus' -- which completely decrypts protected XML on Axis2 and JBoss systems. 

The W3C response: "We are so sorry for promulgating this terrible standard and recommend that you never, ever listen to anything we say regarding security every again." No, of course I'm just kidding. Actually, they're adding AES-GCM as an option. Should take care of everything.
  

·      Quantum key distribution still a work in progress. Quantum Key Distribution (QKD), sometimes known as 'quantum crypto' is a promising technique for distributing keys over geographic distances without the need to rely on complicated stuff like hard mathematical problems. Instead, you just need to run direct fiber-optic cables to the person you want to communicate with (much easier). On the bright side, the security of QKD is supposedly based on fundamental quantum-physical properties.

In general, the caveat in any QKD design is that security only holds if the physical device works as expected. This year researchers from Singapore and Norway showed that this is a big assumption: by exploiting certain limitations of existing detector equipment, they were able to extract an entire secret key without anyone being the wiser. None of these attacks are fundamental 'breaks' of QKD, and indeed they've already been mitigated. But it goes to show how you should never trust a system just because someone tells you it's 'theoretically unbreakable'.

Using Find and Grep

find and grep are incredibly powerful. But many never learn a fraction of what they can do. Did you know that grep can print the surrounding lines? Or that you can search for files based on permissions? Here’s a collection of examples that might expand your notion of what can be done with find and grep.

Most find commands follows the pattern:
$ find [PATH] [expression]

where the expression consists of one or more of the options shown below. find is recursive by default so the command $ find . would output the entire directory tree at the current location.

Find files named Account.java in src directory:
$ find src -name Account.java

Find case insensitive in current directory — matches README, readme, ReadMe, etc.
$ find . -iname readme

Find by wildcard name — NB: quotes keep shell from expanding wildcards, it is best to always include them.
$ find /etc -name '*xml'

Some exclusions — -not can be replaced with !
$ find . -not -name '*java' -maxdepth 4

Find with or
$ find . -name '*java' -o -name '*xml'

There are several ways to do stuff with the results of your find.  I prefer to use the more intuitive xargs rather than -exec, as the -exec syntax is strange. But -exec gives you the ability to choose whether you want the second command executed once (+) or once per result (\;).
$ find . –name '*.java' | xargs wc –l | sort
$ find . –name '*.java' -exec wc –l {} \; | sort
$ find . –name '*.java' -exec wc –l {} + | sort

It is often useful to filter by type. Using find is more reliable than parsing ls.
$ find . -type f
$ find . -type d
$ find . -type l

To find all directories in current directory:
$ find . -type d -maxdepth 1

Do you want to see the files that have changed today?
$ find . -mtime -1

Or the files that have changed in the last 15 minutes?
$ find . -mmin -15

Sometimes it is convenient to find the files that are newer (or older) than a certain file.
$ find . -newer foo.txt
$ find . ! -newer foo.txt

Of the files that have been modified after a certain date … or between two dates.

$ find . -type f -newermt '2010-01-01'
$ find . -type f -newermt '2010-01-01' ! -newermt '2010-06-01'

You can also filter by size. Note that the - finds smaller files, + finds larger files.
$ find . -size -1k
$ find . -size +100M

I don’t know that I’ve every included permissions in a search, but it could be useful.
$ find . -perm 644
$ find . -perm -u=w
$ find . -perm -ug=x

A grep command typically takes the form
$ grep [options] [string/regex] [file or path]

For example:
$ grep 'new Account' *.java

Note that you do not quote a wildcard expression for grep, in this case we want the shell to expand the expression.

grep is not recursive by default, this is altered with the -r flag.
$ grep -r 'Dao[Impl|Mock]' src

I use the following flags commonly:
-i Case insensitive
-w Restricts to word matches
-n outputs line number
-c outputs count of matches

Grepping for multiple terms is useful in theory, but I rarely do it:
$ grep -e foo -e bar baz.txt

Whether searching through source code or log files, the ability to display the surrounding lines is often useful. For example:
$ grep -r -A 2 foo src

will display the lines that contain foo and the two subsequent lines. Similarly, you can use
-B before
-C centered (lines both before and after found text.)

That’s all for now, I hope that this helps you find something soon.