Map Reduce with Bash - An Example of the Unix Philosophy in Action

How used simple bash and Linux commands to build a map/reduce solution for a University's physics department that could be maintained by lecturers and students using existing skills. A stunning example of how the architectural approach of Unix enables users to design simple solutions to their complex processes. The presentation discusses some of the command line utilities used such as GNU parallels and xargs as well how cgroups, namespaces and Linux capabilities can be combined to create a bash based map/reduce framework.
of 18
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Related Documents
  • 1. Jumping Bean Map Reduce With Bash (the power of the Unix philosophy)
  • 2. Jumping Bean About Me ● Solutions integrator at Jumping Bean – Developer & Trainer – Technologies ● Java ● PHP ● HTML5/Javascript ● Linux – What I am planning to do: ● The Internet of things
  • 3. Jumping Bean Map/Reduce with Bash ● Purpose of this presentation is: – to demonstrate the power and flexibility of the Unix philosophy, – what awesome solutions can be created by using simple bash script and userland tools, – cool utilities and tools ● The purpose is not: – to suggest that Map/Reduce is best done with bash – best given constraint – see business problem
  • 4. Unix Philosophy “is a set of cultural norms and philosophical approaches to developing small yet capable Jumping Bean software” - Wikipedia
  • 5. Jumping Bean Unix Philosophy “Early Unix developers were important in bringing the concepts of modularity and reusability into software engineering practice, spawning a 'software tools' movement” - Wikipedia
  • 6. Jumping Bean Business Problem ● Nuclear Engineering department needs to run monte-carlo methods on data to calculate something to do with core temperature of nuclear reactors :), ● Post-grad students need to run analysis as part of their course work, ● Analysis can take days or weeks to run, ● University has invested in 900 node cluster, ● Cluster used for research when not used by students ● Tool used for analysis is – written in Fortran. – single threaded, ● No money for fancy-pants solution
  • 7. Jumping Bean Business Problem ● As-Is system – Professor uses laptop and desktop, – Manually starts application with simple script, – Start script x number of times where x=number of cores, – Waits for days, – Manually checks progress, – Not scalable to 900 nodes!
  • 8. Jumping Bean Business Problem ● Unknowns – How 900 node cluster set up i.e using any cluster software or virtualisation? ● Open Stack? ● Open Nebula? ● KVM? – Tools available to IT department – I.e how they do deploys, monitoring, user management etc ● Requirements – independence from IT department or experts for help, – Student & lecturer IT skills is limited to Fortran & some bash scripting skills, – Due to security concerns prevent IT staff from gaining access to research, ● Keep it simple – Proof of concept
  • 9. Jumping Bean What is Map/Reduce? ● Programming model for – Processing and generating large datasets, – Using a parallel distribution algorithm, – On a cluster or set of distributed nodes ● Popularised by Google and the advent of cloud computing ● Apache Hadoop – full blown map/reduce framework. Used to analyse your social media data, “understand the customer” and by numerous agencies with 3 letter acronyms. – “Really we only trying to help you know yourself better”
  • 10. Jumping Bean Map/Reduce Steps ● Map – Master node takes large dataset and distributes it to compute nodes to perform analysis on. The compute nodes return a result, ● Reduce – Gather the results of the compute nodes and aggregate results into final answer
  • 11. Jumping Bean What we need ● Controller node functions – to distribute data to nodes, – execute calculation functions – collect results ● Management node functions – distribute application and scripts to compute nodes, ● Compute node functions – Scripts to run the single threaded application in parallel on multi core processors ● Security Requirements – Prevent system administrators from gaining access to core application , script or data
  • 12. Jumping Bean Controller Functions ● How to distribute files to a node (map), execute calculations & gather results (reduce)? – Use split to split input files, – Use ssh to distribute files, execute processes, ● How to do this to multiple (900) nodes? – Use parallel ssh (pssh), paralle scp, ● Issues: – Copying public key to 900 machines? – Give each student their own account? ● Solution – Set up ldap authentication (password based) or – Include controller nodes root public key in compute node image, distribute 2ndary keys via scripts using pssh – Fancy pants – chef, ansible
  • 13. Management Node Functions ● Use parallel ssh to distribute scripts from management node to compute nodes, ● Using Ansible or Chef could be a next evolutionary step to automate system maintenance Jumping Bean
  • 14. Jumping Bean Compute Node Functions ● Basically bash scirpt - How to parallelise single threaded application to use multiple cores on modern CPUs? ● xargs – pass through list of input files, – -n set each iteration to run on one input file – -P set number of processes to start in parallel – Script waits for completion of processing & check output ● GNU parallels – Can run commands in parallel using 1 or more hosts – More options for target input placement {}, string replacement – Can pass output as input to another process
  • 15. Jumping Bean Compute/Controller Node ● At end of compute node process either compute node pings controller node, ● Controller node waits for pssh to return to carry out next step. I.e – reduce process or start next script with output from 1st being input to 2nd step, ● Check for errors and reschedule failed computes,
  • 16. Jumping Bean Security ● Each student should have separate account – Linux mutli-user system. User home directory for storing files and results ● Each user should be limited in resource usage – Simple ● ulimit ● psacct – Advanced ● Cgroups ● Namespaces ● Students can execute but not read bash script file, special permissions – Use sudo or – Linux capabilities ● setcap – eg setcap "cap_kill=+ep"
  • 17. Jumping Bean Security ● Limit the root user – Linux capabilities ● setcap, capsh,pscap ● Disable root account – grant CAP_SYS_ADMIN as needed, ● /etc/security/capabilities.conf
  • 18. Jumping Bean Resources ● Parallel SSH, ● Xargs, ● GNU parallel, ● cgroups, ● namespaces, ● Linux capabilities ● Twitter - @mxc4 ● Gplus – Mark Clarke ● Jumping Bean ● Cyber Connect ● Jozi Linux User Grou p ● Jozi Java User Group ● Maker Labs
  • Search
    Similar documents
    View more...
    Related Search
    We Need Your Support
    Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

    Thanks to everyone for your continued support.

    No, Thanks

    We need your sign to support Project to invent "SMART AND CONTROLLABLE REFLECTIVE BALLOONS" to cover the Sun and Save Our Earth.

    More details...

    Sign Now!

    We are very appreciated for your Prompt Action!