Introduction<\/b><\/p>\n
These days I spend a lot of time in the bash shell. I use it for ad-hoc scripting or driving several Linux boxes. In my current project we set up a continuous delivery<\/a> environment and migrate code onto it. I lift code from CVS to SVN, mavenize Ant builds and funnel artifacts into Nexus. One script I wrote determines if a jar that was checked into a CVS source tree exists in Nexus or not. This check can be done via the Nexus REST API<\/a>. More on this script at the end of the blog. But first let’s have a look at a few bash commands that I use all the time in day-to-day bash usage, in no particular order.<\/p>\n Find searches files recursively in the current directory.<\/p>\n This command lists all jars in the current directory, recursively. We use this command to figure out if a source tree has jars. If this is the case we add them to Nexus and to the pom as part of the migration from Ant to Maven.<\/p>\n Find combined with exec is very powerful. This command lists the jars and computes sha1sum for each of them. The shasum command is put directly after the -exec flag. The {} will be replaced with the jar that is found. The \\; is an escaped semicolon for find to figure out when the command ends.<\/p>\n For loops are often the basis of my shell scripts. I start with a for loop that just echoes some values to the terminal so I can check if it works and then go from there.<\/p>\n The for loop keywords should be followed by either a newline or an ‘;’. When the for loop is OK I will add more commands between the do and done blocks. Note that I could have also used find -exec but if I have a script that is more than a one-liner I prefer a for loop for readability.<\/p>\n Transliterate. You can use this to get rid of certain characters or replace them, piecewise.<\/p>\n $ echo ‘Com_Acme_Library’ | tr ‘_A-Z’ ‘.a-z’<\/p>\n Lowercases and replaces underscores with dots.<\/p>\n Prints the second and third column of the output. Awk is of course a full blown programming language but I tend to use this snippets like this a lot for selecting columns from the output of another command.<\/p>\n Stream EDitor. A complete tool on its own, yet I use it mostly for small substitutions.<\/p>\n Replaces foo with quux.<\/p>\n Run a command on every line of input on standard in.<\/p>\n Run sha1sum on every line in the file. This is another for loop or find -exec alternative. I use this when I have a long pipeline of commands in a oneliner and want to process every line in the end result.<\/p>\n Here are some grep features you might not know:<\/p>\n This will list the match of the keyword in data.txt including 3 lines after (-A3) and 3 lines before (-B3) the match.<\/p>\n Inverse match. Match everything except keyword.<\/p>\n Sort is another command often used at the end of a pipeline. For numerical sorting use<\/p>\n This one isn’t a real command but it’s really useful. Instead of typing history and looking up a previous command, press CTRL-R, Pronounced ‘bang-bang’. Repeats the previous command. Here is the cool thing:<\/p>\n This repeats the previous command, but with foo replaced by bar. Useful if you entered a long command with a typo. Instead of manually replacing one of the arguments replace it this way.<\/p>\n Bash script – checking artifacts in Nexus<\/b><\/p>\n Below is the script I talked about. It loops over every jar and dll file in the current directory, calls Nexus via wget and optionally outputs a pom dependency snippet. It also adds a status column at the end of the output, either an OK or a KO, which makes the output easy to grep for further processing.<\/p>\n Conclusions<\/b><\/p>\n It is amazing what you can do in terms of scripting when you combine just these commands via pipes and redirection! It’s like a Pareto’s law<\/a> of shell scripting, 20% of the features of bash and related tools provide 80% of the results. The basis of most scripts can be a for loop. Inside the for loop the resulting data can be transliterated, grepped, replaced by sed and finally run through another program via xargs.<\/p>\n References<\/b><\/p>\n\n
$ find -name *.jar<\/code><\/p>\n$ find -name *.jar -exec sha1sum {} \\;<\/code><\/p>\n
\n$ for i in $(cat items.txt); do echo $i; done;
\n<\/code><\/p>\n
\n$ echo 'one two three' | awk '{ print $2, $3 }'
\n<\/code><\/p>\n
\n$ cat 'foo bar baz' | sed -e 's\/foo\/quux\/'
\n<\/code><\/p>\n
\n$ cat jars.txt | xargs -n1 sha1sum
\n<\/code><\/p>\n$ grep -A3 -B3 keyword data.txt<\/code><\/p>\n$ grep -v keyword data.txt<\/code><\/p>\n$ sort -n<\/code><\/p>\n
\nstart typing and have bash autocomplete your history. Use escape to quit reverse search mode. When you press CTRL-R your prompt will look like this:<\/p>\n(reverse-i-search)`': <\/code><\/p>\n$ !!:s\/foo\/bar<\/code><\/p>\n\n#!\/bin\/bash\n\nok=0\njars=0\n\nfor jar in $(find $(pwd) 2&>\/dev\/null -name '*.jar' -o -name '*.dll')\ndo\n((jars+=1))\n\noutput=$(basename $jar)-pom.xml\nsha1=$(sha1sum $jar | awk '{print $1}')\n\nresponse=$(curl -s http:\/\/oss.sonatype.org\/service\/local\/data_index?sha1=$sha1)\n\nif [[ $response =~ groupId ]]; then\n((ok+=1))\necho "findjars $jar OK"\necho "" >> "$output"\necho "$response" | grep groupId -A3 -m1 >> "$output"\necho "" >> "$output"\nelse\necho "findjars $jar KO"\nfi\n\ndone\n\nif [[ $jars > 0 ]]; then\necho "findjars Found $ok\/$jars jars\/dlls. See -pom.xml file for XML snippet"\nexit 1\nfi\n<\/pre>\n