conversations.from: human
conversations.value: What could be a title for a book about Apache, Hadoop, strong>the with the genre: Computers & Technology, Computer Science?
conversations.from: assistant
conversations.value: A title for this book could be 'Practical Graph Analytics with Apache Giraph'
conversations.from: human
conversations.value: the first five paragraphs for this book
conversations.from: assistant
conversations.value: Practical Graph Analytics with Apache Giraph
  This particular issue is not specific to Giraph; it is known as combinatorial explosion of dependencies: the number of permutations of various versions of dependencies grow exponentially with the number of dependencies.
  So far, the software engineering community has developed two different strategies for dealing with the combinatorial explosion of dependencies:
  Before you deep-dive into the detailed descriptions of these strategies, let's consider the fact that the first strategy is really a subset of the second one. The decision of building your Giraph installation from source effectively means that you're embarking on a mission of producing your own binary software distribution with exact versions of components deployed to your cluster.
  It is interesting to note that, in general, Apache Software Foundation tries to stay away from binary distributions of its projects, leaving this responsibility to downstream packagers. For the established projects (e.g., Apache HTTP server), these packagers are typically distributors of the various operating systems (Linux, etc.). OS distributors make sure that all of the various ASF software projects that end up in their particular version of an OS distribution work smoothly with each other. Hadoop and its ecosystem projects, however, haven't been on the agenda of operating system packagers.
  To fill this void, various commercial vendors of Hadoop distributions began offering fully integrated sets of packages that don't require recompilation and are known to work well with each other. Most of these commercial distributions are based on the work done at Apache Bigtop: a 100% open source, community-driven Big Data management distribution of Apache Hadoop. For most of the users of Hadoop and its ecosystem projects, installing a fully integrated distribution from either Apache Bigtop or a commercial vendor is the easiest way to get started. For those with existing clusters, compiling Giraph from source code to match the exact versions of dependencies deployed on the cluster may be the only option.
conversations.from: human
conversations.value: five more paragraphs
conversations.from: assistant
conversations.value: There is no denying that installing from Linux packages is by far the easiest way to install both Giraph and Hadoop. If you choose one of the commercial vendors, you can sign up for professional support. The downside, however, is that by installing the prepackaged bits, you are giving up your right to have a precise combination of Giraph and Hadoop versions that are unique to your environment. If you find yourself needing to match Giraph to the custom versions of Hadoop and Hadoop ecosystems components, installing Giraph by building from source code may be your only option.
  The source code of Giraph is packaged as a gzipped tar file and available for at the same project web site that you used for downloading binaries: http://giraph.apache.org/releases.html. For the Giraph version 1.1.0, get the file named giraph-dist-1.1.0-src.tar.gz and unpack it somewhere on your development workstation.
  The Giraph build infrastructure is managed by Apache Maven. If you decide to install Giraph by building it from the source code, you have to make sure that Maven version 3 or higher is available in your environment. If you don't have Maven available, make sure to follow installation instructions provided by the project's web site at http://maven.apache.org. The rest of this appendix assumes that you have set up Maven and that you can successfully run the mvn command-line utility.
  You will install Giraph by building it from source code in situations where you have a preexisting Hadoop cluster with the versions of dependencies (including Hadoop itself) not matching those of packaged Giraph binaries. Although building Giraph against arbitrary versions of dependencies is possible, be warned that it may very well be that such a combination has never been tested.
  Giraph's build infrastructure provides a convenient way for specifying exact version of dependencies that Giraph needs to be built for. You don't need to download any of these dependencies or otherwise make them available in your environment--Maven does it for you. You are expected to use Maven build profiles for specifying the major version of Hadoop and when you want to build Giraph as a YARN client rather than a map-only MapReduce application. The most commonly used build profiles are summarized in Table A-3.
conversations.from: human
conversations.value: the last five paragraphs
conversations.from: assistant
conversations.value: At this point, you should have a fully functional HDFS. You can verify that it is up and running by issuing the commands shown in Listing A-8.
  As long as all the commands ran without an error, you can be sure your pseudo-distributed HDFS setup is fine. The final step in setting up your pseudo-distributed Hadoop cluster is starting MapReduce services (for Hadoop 1.x) or YARN services (for Hadoop 2.x). The commands needed are summarized in Tables A-8 and A-9, respectively.
  Regardless of whether you are setting up Hadoop 1.x or Hadoop 2.x, once you are done with these commands, make sure to run a test MapReduce job while you are still located in the current working directory (where the last cd command put you). If the command shown in Listing A-9 runs successfully to completion, it means your pseudo-distributed Hadoop cluster is fully set and ready for your Giraph applications.
  You can install Hadoop and Giraph in a number of different ways. The key issue to keep in mind is that whichever version of Hadoop you install, it has to match the version Giraph binary that you will be using. If you plan to use I/O formats connecting Giraph to other members of the Hadoop ecosystem (such as Hive, HBase, etc.), those versions have to match as well. After installation is done, the last step is to configure both. Giraph doesn't require any configuration by default. Hadoop has to be configured to run in one of the three modes: local, fully distributed, or pseudo-distributed. In this appendix you looked at the following topics:
  Although this appendix offered a number of alternative options for accomplishing two basic tasks-- installation and configuration, if you're installing everything from scratch, you can simply choose whichever is the easiest for your environment. If, however, you are dealing with existing Hadoop clusters, you may find some of the alternative methods helpful.
conversations.from: human
conversations.value: Delete the last sentence.
conversations.from: assistant
conversations.value: At this point, you should have a fully functional HDFS. You can verify that it is up and running by issuing the commands shown in Listing A-8.
  As long as all the commands ran without an error, you can be sure your pseudo-distributed HDFS setup is fine. The final step in setting up your pseudo-distributed Hadoop cluster is starting MapReduce services (for Hadoop 1.x) or YARN services (for Hadoop 2.x). The commands needed are summarized in Tables A-8 and A-9, respectively.
  Regardless of whether you are setting up Hadoop 1.x or Hadoop 2.x, once you are done with these commands, make sure to run a test MapReduce job while you are still located in the current working directory (where the last cd command put you). If the command shown in Listing A-9 runs successfully to completion, it means your pseudo-distributed Hadoop cluster is fully set and ready for your Giraph applications.
  You can install Hadoop and Giraph in a number of different ways. The key issue to keep in mind is that whichever version of Hadoop you install, it has to match the version Giraph binary that you will be using. If you plan to use I/O formats connecting Giraph to other members of the Hadoop ecosystem (such as Hive, HBase, etc.), those versions have to match as well. After installation is done, the last step is to configure both. Giraph doesn't require any configuration by default. Hadoop has to be configured to run in one of the three modes: local, fully distributed, or pseudo-distributed. In this appendix you looked at the following topics:
  Although this appendix offered a number of alternative options for accomplishing two basic tasks-- installation and configuration, if you're installing everything from scratch, you can simply choose whichever is the easiest for your environment.