<?xml version="1.0"?>

<!DOCTYPE document PUBLIC "-//APACHE//DTD Documentation V2.0//EN" 
          "http://forrest.apache.org/dtd/document-v20.dtd">

<!--
 Licensed to the Apache Software Foundation (ASF) under one or more
 contributor license agreements.  See the NOTICE file distributed with
 this work for additional information regarding copyright ownership.
 The ASF licenses this file to You under the Apache License, Version 2.0
 (the "License"); you may not use this file except in compliance with
 the License.  You may obtain a copy of the License at

     http://www.apache.org/licenses/LICENSE-2.0

 Unless required by applicable law or agreed to in writing, software
 distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
-->

<document> 

  <header> 
    <title>Building ManifoldCF</title> 
  </header> 

  <body> 
    <section>
      <title>Building ManifoldCF</title>
      <p></p>
      <p>ManifoldCF consists of a framework, a set of connectors, and an optional Apache2 plug-in module.  These can be built as follows.</p>
      <p></p>
      
      <section>
        <title>Building overview</title>
        <p></p>
        <p>There are two ways to build ManifoldCF.  The primary means of building (and the most supported) is via Apache Ant.  The ant build is used to
          create ManifoldCF releases and to run tests, load tests, and UI tests. Maven is also supported for develop building only.  Maven ManifoldCF builds have
          many restrictions and challenges and are of secondary priority for the development team.</p>
        <p>The ManifoldCF framework is built without any dependencies on connector code.  It consists of a set of jars, a family of web applications, and
          a number of java command classes.  Connectors are then built that have well-defined dependencies on the framework
          modules.  A properly built connector typically consists of:</p>
        <p></p>
        <ul>
           <li>One or more jar files meant to be included in the library area meant for connector jars and their dependencies.</li>
           <li>Possibly some java commands, which are meant to support or configure the connector in some way.</li>
           <li>Possibly a connector-specific process or two, each requiring a distinct classpath, which usually serves to isolate the <strong>crawler-ui</strong> servlet,
            <strong>authority-service</strong> servlet, <strong>agents</strong> process, and any commands from problematic aspects of the client environment</li>
           <li>A recommended set of java "define" variables, which should be used consistently with all involved processes, e.g. the <strong>agents</strong> process, the
            application server running the <strong>authority-service</strong> and <strong>crawler-ui</strong>, and any commands.  (This is historical, and no connectors
            as of this writing have any of these any longer).</li>
        </ul>
        <p></p>
        <p>An individual connector package will typically supply an output connector, or a transformation connector, or a mapping connector, or a repository connector, or
          sometimes both a repository connector and an authority connector.  The
          main ant build script automatically forms each individual connector's contribution to the overall system into the overall package.</p>
      </section>
      
      <section>
        <title>Building the framework and the connectors using Apache Ant</title>
        <p></p>
        <p>To build the ManifoldCF framework code, and the particular connectors you are interested in, you currently need to do the following:</p>
        <p></p>
        <ol>
          <li>Check out the desired release from https://svn.apache.org/repos/asf/manifoldcf/tags, or unpack the desired source distribution.</li>
          <li>cd to the top-level directory.</li>
          <li><strong>EITHER:</strong> overlay the lib directory from the corresponding lib distribution (preferred, where possible), <strong>OR</strong> run "ant make-core-deps" to
            build the code dependencies.  The latter is the only possibility if you are building from trunk, but it is not guaranteed to work for older releases.</li>
          <li>Run "ant make-deps", to download LGPL and other open source but non-Apache compatible libraries.</li>
          <li>Install proprietary build dependencies.  See below for details.</li>
          <li>Run "ant build".</li>
          <li>Install desired dependent proprietary libraries.  See below for details.</li>
        </ol>
        <p></p>
        <p>If you do not run the ant "make-deps" target, and you supply <strong>NO</strong> LGPL or proprietary libraries, not all capabilities of ManifoldCF will be available.
            The framework itself and the following repository connectors will be built:</p>
        <p></p>
        <ul>
          <li>Alfresco Webscript connector</li>
          <li>Amazon S3 connector</li>
          <li>CMIS connector</li>
          <li>Atlassian Confluence connector</li>
          <li>EMC Documentum connector, built against a Documentum API stub</li>
          <li>DropBox connector</li>
          <li>Email connector</li>
          <li>FileNet connector, built against a FileNet API stub</li>
          <li>WGET-compatible filesystem connector</li>
          <li>Generic XML repository connector</li>
          <li>Google Drive connector</li>
          <li>GridFS connector (mongoDB)</li>
          <li>HDFS connector</li>
          <li>JDBC connector, with the just the POSTGRESQL jdbc driver</li>
          <li>Atlassian Jira connector</li>
          <li>OpenText LiveLink connector, built against a LiveLink API stub</li>
          <li>Meridio connector, built against modified Meridio API WSDLs and XSDs</li>
          <li>RSS connector</li>
          <li>Microsoft SharePoint connector, built against SharePoint API WSDLs</li>
          <li>Webcrawler connector</li>
          <li>Wiki connector</li>
        </ul>
        <p></p>
        <p>The following authority connectors will be built:</p>
        <p></p>
        <ul>
          <li>Active Directory authority</li>
          <li>Alfresco Webscript authority</li>
          <li>Amazon S3 authority</li>
          <li>CMIS authority</li>
          <li>Atlassian Confluence authority</li>
          <li>EMC Documentum authority</li>
          <li>Atlassian Jira authority</li>
          <li>LDAP authority</li>
          <li>OpenText LiveLink authority</li>
          <li>Meridio authority, built against modified Meridio API WSDLs and XSDs</li>
          <li>Null authority</li>
          <li>Microsoft SharePoint/AD authority</li>
          <li>Microsoft SharePoint/Native authority, built against SharePoint API WSDLs</li>
        </ul>
        <p></p>
        <p>The following output connectors will be built:</p>
        <p></p>
        <ul>
          <li>Amazon Cloud Search output connector</li>
          <li>WGET-compatible filesystem output connector</li>
          <li>MetaCarta GTS output connector</li>
          <li>Apache Solr output connector</li>
          <li>OpenSearchServer output connector</li>
          <li>ElasticSearch output connector</li>
          <li>WGET-compatible filesystem output connector</li>
          <li>HDFS output connector</li>
          <li>Kafka output connector</li>
          <li>Null output connector</li>
          <li>Searchblox output connector</li>
        </ul>
        <p></p>
        <p>The following transformation connectors will be built:</p>
        <p></p>
        <ul>
          <li>Field mapping transformation connector</li>
          <li>Document filter transformation connector</li>
          <li>OpenNLP transformation connector</li>
          <li>Null transformation connector</li>
          <li>Tika extractor transformation connector</li>
        </ul>
        <p></p>
        <p>The following mapping connectors will be built:</p>
        <p></p>
        <ul>
          <li>Regular-expression mapping connector</li>
        </ul>
        <p></p>
        <p>The following notification connectors will be built:</p>
        <p></p>
        <ul>
          <li>Email notification connector</li>
        </ul>
        <p></p>
        <p>The dependencies and build limitations of each individual LGPL and proprietary connector is described in separate sections below.</p>
        <p></p>
            
        <section>
          <title>Building and testing the legacy Alfresco connector</title>
          <p></p>
          <p>The legacy Alfresco connector requires the Alfresco Web Services Client provided by Alfresco in order to be built. Place this jar into the directory <em>connectors/alfresco/lib-proprietary</em> before you build.
              This will occur automatically if you execute the ant target "make-deps" from the ManifoldCF root directory.</p>
          <p></p>
          <p>To run integration tests for the connector you have to copy the alfresco.war including H2 support created by the Maven module test-materials/alfresco-4-war (using "mvn package" inside the folder)
              into the <em>connectors/alfresco/test-materials-proprietary</em> folder.  Then use the "ant test" or "mvn integration-test" for the standard build to execute integration tests.</p>
          <p></p>
        </section>

        <section>
          <title>Building and testing the Alfresco Webscript connector</title>
          <p></p>
          <p>The Alfresco Webscript connector is built against an open-source Alfresco Indexer client, which requires a corresponding Alfresco Indexer plugin
                to be installed on your Alfresco instance.  This Alfresco Indexer plugin is included with ManifoldCF distributions.  Installation of the plugin should follow the
                standard Alfresco installation steps, as described <a href="http://docs.alfresco.com/4.1/tasks/amp-install.html">here</a>.
                See <a href="https://github.com/maoo/alfresco-indexer">this page</a> for configuration details, and for the plugin itself.</p>
          <p></p>
        </section>

        <section>
          <title>Building and running the Documentum connector</title>
          <p></p>
          <p>The Documentum connector requires EMC's DFC product in order to be run, but is built against a set of stub classes.
              The stubs mimic the class structure of DFC 6.0.  If your DFC is newer, it is possible that the class structure of the DFC
              classes might have changed, and you may need to build the connector yourself.</p>
          <p>If you need to supply DFC classes during build time, copy the DFC and dependent jars to the source directory
              <em>connectors/documentum/lib-proprietary</em>, and build using "ant build".  The jars will be copied  into
              the right place in your <em>dist</em> directory automatically.</p>
          <p>For a binary distribution, just copy the DFC jars to <em>processes/documentum-server/lib-proprietary</em> instead.</p>
          <p>If you have done everything right, you should be able to start the Documentum connector's registry and server processes, as per the instructions.</p>
          <p></p>
        </section>
        
        <section>
          <title>Building and running the FileNet connector</title>
          <p></p>
          <p>The FileNet connector requires IBM's FileNet P8 API jar in order to be run, but is usually built against a set of stub classes.
              The stubs mimic the class structure of FileNet P8 API 4.5.  If your FileNet is newer, it is possible that the class structure of the
              API might have changed, and you may need to build the connector yourself.</p>
          <p>If you need to supply your own Jace.jar at build time, copy it to the source directory <em>connectors/filenet/lib-proprietary</em>,
              and build using "ant build".  The Jace.jar will be copied into the right place in your <em>dist</em> directory automatically.</p>
          <p>If you do not wish to build, simply copy your Jace.jar and the other dependent jars from that installation into the distribution directory
              <em>processes/filenet-server/lib-proprietary</em>.</p>
          <p>If correctly done, you will be able to start the FileNet connector's registry and server processes, as per the instructions.</p>
          <p></p>
        </section>
        
        <section>
          <title>Building and running the JDBC connector, including Oracle, MSSQL, MySQL, SQLServer, and Sybase JDBC drivers</title>
          <p></p>
          <p>The JDBC connector also knows how to work with Oracle, SQLServer, and Sybase JDBC drivers.  In order to support these databases, start by placing the mysql-connector-java.jar
            and the jtds.jar in the <em>lib-proprietary</em> directory.  The ant target "make-deps" will do this for you automatically.  For Oracle, download the appropriate
            Oracle JDBC jar from the Oracle site, and copy it into the same directory before you build ManifoldCF.</p>
          <p></p>
        </section>
        
        <section>
          <title>Building and running the jCIFS/Windows Shares connector</title>
          <p></p>
          <p>To build this connector, you need to download jcifs.jar from <a href="http://jcifs.samba.org">http://jcifs.samba.org</a>, and copy it into the <em>connectors/jcifs/lib-proprietary</em>
              directory before building.  You can also just type "ant make-deps" from the root ManifoldCF directory and this step will be done for you.</p>
          <p>If you have downloaded a binary distribution, place the jcifs.jar into the <em>connector-lib-proprietary</em> directory, and uncomment the Windows Shares line in the <em>connectors.xml</em> file.</p>
          <p></p>
        </section>
        
        <section>
          <title>Building and running the LiveLink connector</title>
          <p></p>
          <p>This connector needs OpenText's LAPI package in order to be run.  It is usually built against a set of stubs. The stubs, however, mimic the
              class structure of LAPI 9.7.1.  Later versions (such as 10.x) have a different class structure.  Therefore, you may need to rebuild ManifoldCF
              against your lapi.jar, in order for the connector to work properly.</p>
          <p>If you need to supply your own lapi.jar and llssl.jar at build time, copy it to the source directory <em>connectors/livelink/lib-proprietary</em>, and build using "ant build".
              The lapi.jar will be copied into the right place in your <em>dist</em> directory automatically.</p>
          <p>If you do not wish to build, simply copy your lapi.jar and llssl.jar into the binary distribution's <em>connector-lib-proprietary</em>
              directory, and uncomment the LiveLink-related connector lines in <em>connectors.xml</em> file.</p>
          <p></p>
        </section>
        
        <section>
          <title>Building the Meridio connector</title>
          <p></p>
          <p>The Meridio connector generates interface classes using checked-in wsdls and xsds originally obtained
              from an installed Meridio instance using <strong>disco.exe</strong>, and subsequently modified to work
              around limitations in Apache Axis.  The <strong>disco.exe</strong> utility is installed as part of
              Microsoft Visual Studio, and is typically found under "c:\Program Files\Microsoft SDKs\Windows\V6.x\bin".
              If desired, you can obtain unmodified wsdls and xsds by interrogating the following Meridio web services:</p>
          <p></p>
          <ul>
             <li>http[s]://&#60;meridio_server&#62;/DMWS/MeridioDMWS.asmx</li>
             <li>http[s]://&#60;meridio_server&#62;/RMWS/MeridioRMWS.asmx</li>
          </ul>
          <p></p>
        </section>
        
        <section>
          <title>Building and running the SharePoint connector</title>
          <p></p>
          <p>The SharePoint connector generates interface classes using checked-in wsdls originally obtained
            from an installed Microsoft SharePoint instance using <strong>disco.exe</strong>.  The <strong>disco.exe</strong> utility is installed as part of
            Microsoft Visual Studio, and is typically found under "c:\Program Files\Microsoft SDKs\Windows\V6.x\bin".
            If desired, you can obtain unmodified wsdls by interrogating the following SharePoint web services:</p>
          <p></p>
          <ul>
             <li>http[s]://&#60;server_name&#62;/_vti_bin/Permissions.asmx</li>
             <li>http[s]://&#60;server_name&#62;/_vti_bin/Lists.asmx</li>
             <li>http[s]://&#60;server_name&#62;/_vti_bin/Dspsts.asmx</li>
             <li>http[s]://&#60;server_name&#62;/_vti_bin/usergroup.asmx</li>
             <li>http[s]://&#60;server_name&#62;/_vti_bin/versions.asmx</li>
             <li>http[s]://&#60;server_name&#62;/_vti_bin/webs.asmx</li>
          </ul>
          <p></p>
          <p><strong>Important:</strong> For SharePoint instances version 3.0 (2007) or higher, in order to run the connector, you also must deploy a custom SharePoint web service
            on the SharePoint instance you intend to connect to.  This is required because Microsoft overlooked support for web-service-based access to file and folder security information
            when SharePoint 2007 was released.  For SharePoint version 4.0 (2010), the service is even more critical, because backwards compatibility was not maintained and without
            this service no crawling can occur.  SharePoint version 5.0 (2013) also requires a plugin; although its functionality is the same as for SharePoint 4.0, the binary you install
            is built against SharePoint 2013 resources rather than SharePoint 2010 resources, so there is a different distribution.</p>
          <p>The versions of this service can be found in the distribution directory <em>plugins/sharepoint</em>.
            Pick the version appropriate for your SharePoint installation, and install it following the instructions in the file <strong>Installation Readme.txt</strong>, found in the
            corresponding directory.</p>
          <p></p>
        </section>

        <section>
          <title>Running the Apache OpenNLP transformation connector</title>
          <p></p>
          <p>The Apache OpenNLP transformation connector requires OpenNLP models in order to be useful.  These models should be downloaded or created by you, and
            must be placed in the <em>file-resources</em> folder.  This will allow the connector to find them and allow you to use them.</p>
          <p>The models available publicly for OpenNLP can be downloaded from this URL: <a href="http://opennlp.sourceforge.net/models-1.5/">http://opennlp.sourceforge.net/models-1.5/</a>.  To use the
            connector, you <strong>must</strong> specify a sentence model and a tokenization model.  You can specify as few or as many content extraction models as you
            wish.</p>
          <p></p>
          <p><strong>If you do not download or otherwise provide models for the OpenNLP connector's use, it will not be possible to define a job which uses the connector.</strong></p>
        </section>

        <section>
          <title>Running the Apache Solr output connector</title>
          <p></p>
          <p>The Apache Solr output connector requires no special attention to build or run within ManifoldCF.  However, in order for Apache Solr to be able to enforce
              document security, you must install and properly configure a plugin for Solr.  This plugin is available for both Solr 3.x and for Solr 4.x, and can be used either
              as a query parser plugin, or as a search component.  Additional index fields are also required to contain the necessary security information.  Much more
              information can be found in the README.txt file in the plugins themselves.</p>
          <p>The correct versions of the plugins are included in the <em>plugins/solr</em> directory of the main ManifoldCF distribution.  You can also download updated versions
              of the plugins from the ManifoldCF download page.  The compatibility matrix is as follows:</p>
          <p></p>
          <table>
            <caption>Apache ManifoldCF Solr 3.x and Solr 4.x plugin compatibility</caption>
            <tr><th>ManifoldCF versions</th><th>Plugin version</th></tr>
            <tr><td>0.1.x-1.4.x</td><td>0.x</td></tr>
            <tr><td>1.5.x</td><td>1.x</td></tr>
            <tr><td>&#62;=1.6.x</td><td>2.x</td></tr>
          </table>
          <p></p>
          <p><strong>If the proper version of the plugin is not deployed on Solr, documents will not be properly secured.</strong>  Thus, it is essential to verify that the
              proper plugin version has been deployed for the version of ManifoldCF you are using.</p>
        </section>

        <section>
          <title>Running the ElasticSearch output connector</title>
          <p></p>
          <p>The ElasticSearch output connector requires no special attention to build or run within ManifoldCF.  However, in order for ElasticSearch to be able to enforce
              document security, you must install, properly configure, and code against a toolkit plugin for ElasticSearch.
              Additional index fields are also required to contain the necessary security information.  Much more information can be found in the README.txt file in the
              plugin itself.</p>
          <p>The correct versions of the plugin is included in the <em>plugins/elasticsearch</em> directory of the main ManifoldCF distribution.  You can also download updated versions
              of the plugin from the ManifoldCF download page.  The compatibility matrix is as follows:</p>
          <p></p>
          <table>
            <caption>Apache ManifoldCF ElasticSearch plugin compatibility</caption>
            <tr><th>ManifoldCF versions</th><th>Plugin version</th></tr>
            <tr><td>0.1.x-1.4.x</td><td>0.x</td></tr>
            <tr><td>1.5.x</td><td>1.x</td></tr>
            <tr><td>&#62;=1.6.x</td><td>2.x</td></tr>
          </table>
          <p></p>
          <p><strong>If the proper version of the plugin is not deployed and properly integrated, documents will not be properly secured.</strong>  Thus, it is essential to verify that the
              proper plugin version has been deployed for the version of ManifoldCF you are using.</p>

          <p>To work with ManifoldCF, your ElasticSearch instance must also include the appropriate indexes created as well.
              Here are some simple steps for creating an ElasticSearch index, using the CURL utility:</p>
          <p></p>
          <source>
% curl -XPUT 'http://localhost:9200/manifoldcf'
% curl -XPUT 'http://localhost:9200/manifoldcf/attachment/_mapping' -d '
{
  "attachment" : {
    "_source" : {
      "excludes" : [ "file" ]
    },
    "properties": { 
      "allow_token_document" : { 
        "type" : "string" 
      },
      "allow_token_parent" : { 
        "type" : "string" 
      },
      "allow_token_share" : { 
        "type" : "string" 
      },
      "attributes" : {
        "type" : "string"
      },
      "createdOn" : {
        "type" : "string"
      },
      "deny_token_document" : {
        "type" : "string"
      },
      "deny_token_parent" : {
        "type" : "string"
      },
      "deny_token_share" : {
        "type" : "string"
      },
      "lastModified" : {
        "type" : "string"
      },
      "shareName" : {
        "type" : "string"
      },
      "file" : {
        "type" : "attachment",
        "path" : "full",
        "fields" : {
          "file" : {
            "store" : true,
            "term_vector" : "with_positions_offsets",
            "type" : "string"
          }
        }
      }
    }
  }
}'
          </source>
          <p>This command creates an index called <code>manifoldcf</code> with a mapping named <code>attachment</code>
                which has some generic fields for access tokens and a field <code>file</code> which makes use of the
                ElasticSearch attachment mapper plugin. It is configured for highlighting (<code>"term_vector" : "with_positions_offsets"</code>).</p>
          <p>The following part is useful for not saving the source json on the index which reduces the index size significantly. Be
                aware that you shouldn't do this if you will need to re-index data on the ElasticSearch side or you need access to the
                whole document:</p>
          <source>
"_source" : {
  "excludes" : [ "file" ]
},
          </source>
        </section>

      </section>
      
      <section>
        <title>Building the framework and the connectors using Apache Maven</title>
        <p></p>
        <p>ManifoldCF includes some support for building jars under Maven.  Apache Ant is considered to be ManifoldCF's primary build system, so your mileage with Maven may vary.</p>
        <p>The biggest limitation of the current Maven build is that it does not support any of the proprietary connectors or the multi-process model of execution.
            The build includes only the Apache-licensed and LGPL-licensed connectors, thus avoiding conditional compilation, and executes under Maven using only the
            Quick Start example.</p>
        <p>A canned version of all configuration files are included as resources.  If you want to change the configuration in any way, you will need to rebuild with Maven accordingly.</p>
        <section>
          <title>Preparation</title>
          <p>No special preparation is required, other than to have access to the Apache Maven repository.</p>
        </section>
        <section>
          <title>How to build</title>
          <p>Building is straightforward.  In the ManifoldCF root, type:</p>
          <source>
mvn clean install
          </source>
          <p>This should generate all the necessary artifacts to run with, and also run the Hsqldb-based tests.</p>
          <p>To build and skip only the integration tests, type:</p>
          <source>
mvn clean install -DskipITs   
          </source>
          <p>When you have the default package installed locally in your Maven repository, to only build ManifoldCF artifacts, type:</p>
          <source>
mvn clean package
          </source>
          <p>NOTE: Due to current limitations in the ManifoldCF Maven poms, you MUST run a complete "mvn clean install" as the first step.  You cannot skip steps, or the build will fail.</p>
        </section>
      </section>
      
      <section>
        <title>Building ManifoldCF's Apache2 plugin</title>
        <p></p>
        <p>To build the mod-authz-annotate plugin, you need to start with a Unix system that has the apache2 development tools installed on it, plus the curl development package
          (from <a href="http://curl.haxx.se">http://curl.haxx.se</a> or elsewhere).  Then, cd to mod-authz-annotate, and type "make".  The build will produce a file called
          mod-authz-annotate.so, which should be copied to the appropriate Apache2 directory so it can be used as a plugin.</p>
        <p></p>
        <p></p>
      </section>
        
    </section>
    
    <section>
      <title>Running ManifoldCF</title>
      <p></p>
      <section>
        <title>Overview</title>
        <p>ManifoldCF consists of several components.  These are enumerated below:</p>
        <p></p>
        <ul>
           <li>A database, which is where ManifoldCF keeps all of its configuration and state information, usually PostgreSQL</li>
           <li>A synchronization directory, which how ManifoldCF coordinates activity among its various processes</li>
           <li>An <strong>agents</strong> process, which is the process that actually crawls documents and ingests them</li>
           <li>A <strong>crawler-ui</strong> servlet, which presents the UI users interact with to configure and control the crawler</li>
           <li>An <strong>authority-service</strong> servlet, which responds to requests for authorization tokens, given a user name</li>
           <li>An <strong>api-service</strong> servlet, which responds to REST API requests</li>
        </ul>
        <p></p>
        <p>These underlying components can be packaged in many ways.  For example, the three servlets can be deployed in separate
          war fields as separate web applications.  One may also deploy all three servlets in one combined web application, and also include the
          agents process.</p>
        <p></p>
      </section>
      <p></p>
      <section>
        <title>Binary organization</title>
        <p>Whether you build ManifoldCF yourself, or download a binary distribution, you will need to know what is what in the build result.  If you build ManifoldCF yourself, the binary build
          result can be found in the subdirectory <em>dist</em>.  In a binary distribution, the contents of the distribution are the contents of the <em>dist</em> directory.
          These contents are described below.</p>
        <p></p>
        <table>
          <caption>Distribution directories and files</caption>
          <tr><th><em>dist</em> file/directory</th><th>Meaning</th></tr>
          <tr><td><em>connectors.xml</em></td><td>an xml file describing the connectors that should be registered</td></tr>
          <tr><td><em>connector-lib</em></td><td>jars for all the connectors, referred to by properties.xml</td></tr>
          <tr><td><em>connector-lib-proprietary</em></td><td>proprietary jars for all the connectors, referred to by properties.xml; not included in binary release</td></tr>
          <tr><td><em>obfuscation-utility</em></td><td>a utility to obfuscate passwords, for inclusion in properties.xml fields</td></tr>
          <tr><td><em>lib</em></td><td>jars for all of the examples, referenced by the example scripts</td></tr>
          <tr><td><em>lib-proprietary</em></td><td>proprietary jars for all of the examples, referenced by the proprietary example scripts</td></tr>
          <tr><td><em>processes</em></td><td>scripts, classpath jars, and -D switch values needed for the required connector-specific processes</td></tr>
          <tr><td><em>script-engine</em></td><td>jars and scripts for running the ManifoldCF script interpreter</td></tr>
          <tr><td><em>example</em></td><td>a jetty-based example that runs in a single process (except for any connector-specific processes), excluding all proprietary libraries</td></tr>
          <tr><td><em>example-proprietary</em></td><td>a jetty-based example that runs in a single process (except for any connector-specific processes), including proprietary libraries; not included in binary release</td></tr>
          <tr><td><em>multiprocess-file-example</em></td><td>scripts and jars for an example that uses the multiple process model using file-based synchronization, excluding all proprietary libraries</td></tr>
          <tr><td><em>multiprocess-file-example-proprietary</em></td><td>scripts and jars for an example that uses the multiple process model using file-based synchronization, including proprietary libraries; not included in binary release</td></tr>
          <tr><td><em>multiprocess-zk-example</em></td><td>scripts and jars for an example that uses the multiple process model using ZooKeeper-based synchronization, excluding all proprietary libraries</td></tr>
          <tr><td><em>multiprocess-zk-example-proprietary</em></td><td>scripts and jars for an example that uses the multiple process model using ZooKeeper-based synchronization, including proprietary libraries; not included in binary release</td></tr>
          <tr><td><em>web</em></td><td>app-server deployable web applications (wars), excluding all proprietary libraries</td></tr>
          <tr><td><em>web-proprietary</em></td><td>app-server deployable web applications (wars), including proprietary libraries; not included in binary release</td></tr>
          <tr><td><em>doc</em></td><td>javadocs for framework and all included connectors</td></tr>
          <tr><td><em>plugins</em></td><td>pre-built integration components to deploy on target systems, e.g. for Solr</td></tr>
          <tr><td><em>file-resources</em></td><td>a directory meant to contain file resources that are identically present on all cluster members, e.g. models for OpenNLP</td></tr>
        </table>
        <p></p>
        <p>If you downloaded the binary distribution, you may notice that the <em>connector-lib-proprietary</em> directory contains only a number of 
            <em>&#60;connector&#62;-README.txt</em> files.
            This is because under Apache licensing rules, incompatibly-licensed jars may not be redistributed.  Each such <em>&#60;connector&#62;-README.txt</em> describes
            the jars that you need to add to the <em>connector-lib-proprietary</em> directory in order to get the corresponding connector working.  You will also then need to uncomment
            the appropriate entries in the <em>connectors.xml</em> file accordingly to enable the connector for use.</p>
        <p></p>
        <p>NOTE: The prebuilt binary distribution cannot, at this time, include support for MySQL.  Nor can the JDBC Connector access MySQL, MSSQL, SyBase, or Oracle databases in that
            distribution.  In order to use these JDBC drivers, you must build ManifoldCF yourself.  Start by downloading the drivers and placing them in the <em>lib-proprietary</em> directory.  The command
            <em>ant download-dependencies</em> will do most of this for you, with the exception of the Oracle JDBC driver.</p>
        <p></p>
        <p>The directory titled <em>processes</em> include separate processes which must be started in order for the associated connector to function.
            The number of produced <em>processes</em> subdirectories may vary, because optional individual connectors may or may not supply processes that
            must be run to support the connector.  For each of the <em>processes</em> subdirectories above, any scripts that pertain to that connector-supplied
            process will be placed in the root level of the subdirectory.
            The supplied scripts for a process generally take care of building an appropriate classpath and setting necessary -D switches.  (Note: none of the current
            connectors require special -D switches at this time.)  If you need to construct a classpath by hand, it is important to remember that "more" is not necessarily
            "better".  The process deployment strategy implied by the build structure has
            been carefully thought out to avoid jar conflicts.  Indeed, several connectors are structured using multiple processes precisely for that reason.</p>
        <p>The proprietary libraries required by the secondary process <em>processes</em> subdirectories should be in the directory <em>processes/xxx/lib-proprietary</em>.
            These jars are not included in the binary distribution, and you will need to supply them in order to make the process work.  A README.txt file is placed
            in each <em>lib-proprietary</em> directory describing what needs to be provided there.</p>
        <p></p>
        <p>The <em>plugins</em> directory contains components you may need to deploy on the target system to make the associated connector function correctly.  For example, the Solr
            connector includes plug-in classes for enforcing ManifoldCF security on Solr 3.x and 4.x.  See the README file in each directory for detailed instructions on how to deploy the components.</p>
        <p></p>
        <p>The <em>file-resources</em> directory contains a README.txt file plus any files your connectors specifically need access to across all cluster members.
            You should have an exact copy of this directory present on every machine that a related ManifoldCF process runs on, and the properties.xml file parameter
            <em>org.apache.manifoldcf.fileresources</em> should point to it.  Any third-party or custom files needed by your connector configuration (e.g. OpenNLP models)
            should be created or downloaded and placed here.</p>
        <p></p>
        <p>Inside the <em>example</em> directory, you will find everything you need to fire up ManifoldCF in a single-process model under Jetty.  Everything is included so that all you need to do is change
            to that directory, and start it using the command <em>&lt;java&gt; -jar start.jar</em>.  This is described in more detail later, and is the recommended way for beginners to try out ManifoldCF.
            The directory <em>example-proprietary</em> contains an equivalent example that includes proprietary connectors and jars.  This is the standard place to start if you build ManifoldCF yourself.</p>
        <p></p>
      </section>

      <section>
        <title>Example deployments</title>
        <p>There are many different ways to run ManifoldCF out-of-the-box.  These are enumerated below:</p>
        <ul>
          <li>Quick-start single process model</li>
          <li>Single-process deployable war</li>
          <li>Simplified multi-process model</li>
          <li>Command-driven multi-process model</li>
        </ul>
        <p>All examples have similar script structure and construction.  For example, there are always options files you can modify to control the behavior of individual processes.  These
          files have names of the form: <em>*-options.env.win</em> and <em>*-options.env.unix</em>.  They contain a list of switches to be used in starting up the JVM.  These
          switches also include memory specification (the <em>-Xmx</em> switch), as well as a predefined <em>-D</em> switch that is meant to enable Kerberos (for the connectors
          it is implemented for).  In general, for <em>-D</em> switches in options files, you will want to change them for all options files in the example deployment you are using, to
          insure that all processes operate in the same environment.  For example, to enable Kerberos, you must change the options files in the following way: the <em>-Djava.security.auth.login.config=</em>
          line should be changed to <em>-Djava.security.auth.login.config=jaas-client.conf</em>, and the <em>jaas-client.conf</em> file filled out properly for your installation.</p>
        <p></p>
        <p>Each example deployment has advantages and disadvantages.  For example, single-process models limit the flexibility of deploying ManifoldCF components.  Multi-process models require that
          inter-process synchronization be properly configured.  If you are just starting out with ManifoldCF, we suggest you try the quick-start single process model first, since that is
          the easiest.</p>
        <section>
          <title>Quick-start single process model</title>
          <p></p>
          <p>You can run most of ManifoldCF in a single process, for evaluation and convenience.  This single-process version uses Jetty to handle its web applications, and Hsqldb as
            an embedded database.  All you need to do to run this version of ManifoldCF is to follow the Ant-based build instructions above, and then:</p>
          <p></p>
          <source>
cd example
start[.bat|.sh]
          </source>
          <p></p>
          <p>In the quick-start model, all database initialization and connector registration takes place automatically whenever ManifoldCF is started (at the cost of some startup delay).
            The crawler UI can be found at http://&#60;host&#62;:8345/mcf-crawler-ui.  The authority service can be found at http://&#60;host&#62;:8345/mcf-authority-service/UserACLs.
            The programmatic API is at http://&#60;host&#62;:8345/mcf-api-service.</p>
          <p></p>
          <p>You can stop the quick-start ManifoldCF at any time using ^C, or by using the script <em>stop[.bat|.sh]</em></p>
          <p></p>
          <p>Bear in mind that Hsqldb is not as full-featured a database as is PostgreSQL.  This means that any performance testing you may do against the quick start example may
            not be applicable to a full installation.  Furthermore, embedded Hsqldb only permits one process at a time to be connected to its databases, so you <strong>cannot</strong> use any
            of the ManifoldCF commands (as described below) while the quick-start ManifoldCF is running.</p>
          <p></p>
          <p>Another caveat that you will need to be aware of with the quick-start version of ManifoldCF is that it in no way removes the need for you to run any separate processes
            that individual connectors require.  Specifically, the Documentum and FileNet connectors require processes to be independently started in order to function.  You will need
            to read about these connector-specific processes below in order to use the corresponding connectors.  Scripts for running these processes can be found in the directories
            named <em>processes/xxx</em>.</p>
          <p></p>
        </section>
        
        <section>
          <title>Single-process deployable war</title>
          <p></p>
          <p>Under the distribution directory <em>web/war</em>, there is a war file called <em>mcf-combined-service.war</em>.  This web application contains the exact same
            functionality as the quick-start example, but bundled up as a single war instead.  An example script is provided to run this web application under Jetty.  You can execute
            the script as follows:</p>
          <p></p>
          <source>
  cd example
  start-combined[.sh|.bat]
          </source>
          <p></p>
          <p>The combined web service presents the crawler UI at the root path for the web application, which is <em>http://&#60;host&#62;:8345/mcf/</em>.  The authority
            service functionality can be found at <em>http://&#60;host&#62;:8345/mcf/UserACLs</em>, similar to the quick-start example.  However, the programmatic API service has a path
            other than the root: <em>http://&#60;host&#62;:8345/mcf/api/</em>.</p>
          <p>The script that starts the combined-service web application uses the same database instance (Hsqldb by default) as does the quick-start, and the same <em>properties.xml</em>
            file.  The same caveats about required individual connector processes also apply as they do for the quick-start example.</p>
          <p></p>

          <section>
            <title>Running single-process combined war example using Tomcat</title>
            <p>In order to run the ManifoldCF single-process combined war example under Tomcat, you will need to take the following steps:</p>
            <p></p>
            <ol>
              <li>Modify the Tomcat startup script, or use the Tomcat service administration client, to set a Java "-Dorg.apache.manifoldcf.configfile" switch to point to the example's <em>properties.xml</em> file.</li>
              <li>Start Tomcat.</li>
              <li>Deploy and start the mcf-combined-service web application, preferably using the Tomcat administration client.</li>
            </ol>
          </section>

        </section>

        <section>
          <title>Simplified multi-process model using file-based synchronization</title>
          <p></p>
          <p>ManifoldCF can also be deployed in a simplified multi-process model which uses files to synchronize processes.  Inside the <em>multiprocess-file-example</em> directory, you will find everything you need to do this.  (The
              <em>multiprocess-file-example-proprietary</em> directory is similar but includes proprietary material and is available only if you build ManifoldCF yourself.)  Below is a list of
              what you will find in this directory.</p>
          <p></p>
          <table>
            <caption>File-based multiprocess example files and directories</caption>
            <tr><th><em>multiprocess-file-example</em> file/directory</th><th>Meaning</th></tr>
            <tr><td><em>web</em></td><td>Web applications that should be deployed on tomcat or the equivalent, plus recommended application server -D switch names and values</td></tr>
            <tr><td><em>processes</em></td><td>classpath jars that should be included in the class path for all non-connector-specific processes, along with -D switches, using the same convention as described for tomcat, above</td></tr>
            <tr><td><em>properties.xml</em></td><td>an example ManifoldCF configuration file, in the right place for the multiprocess script to find it</td></tr>
            <tr><td><em>logging.xml</em></td><td>an example ManifoldCF logging configuration file, in the right place for the properties.xml to find it</td></tr>
            <tr><td><em>syncharea</em></td><td>an example ManifoldCF synchronization directory, which must be writable in order for multiprocess ManifoldCF to work</td></tr>
            <tr><td><em>logs</em></td><td>where the ManifoldCF logs get written to</td></tr>
            <tr><td><em>start-database[.sh|.bat]</em></td><td>script to start the HSQLDB database</td></tr>
            <tr><td><em>initialize[.sh|.bat]</em></td><td>script to create the database instance, create all database tables, and register connectors</td></tr>
            <tr><td><em>start-webapps[.sh|.bat]</em></td><td>script to start Jetty with the ManifoldCF web applications deployed</td></tr>
            <tr><td><em>start-agents[.sh|.bat]</em></td><td>script to start the (first) agents process</td></tr>
            <tr><td><em>start-agents-2[.sh|.bat]</em></td><td>script to start a second agents process</td></tr>
            <tr><td><em>stop-agents[.sh|.bat]</em></td><td>script to stop all running agents processes cleanly</td></tr>
            <tr><td><em>lock-clean[.sh|.bat]</em></td><td>script to clean up dirty locks (run only when all webapps and processes are stopped)</td></tr>
          </table>
          <p></p>
          <section>
            <title>Initializing the database and running</title>
            <p></p>
            <p>If you run the file-based multiprocess model, after you first start the database (using <em>start-database[.sh|.bat]</em>), you will need to initialize the database before you start the agents process or use the crawler UI.  To do this, all you need to do is
                run the <em>initialize[.sh|.bat]</em> script.  Then, you will need to start the web applications (using <em>start-webapps[.sh|.bat]</em>) and the agents process (using
                <em>start-agents[.sh|.bat]</em>), and optionally the second agents process (using <em>start-agents-2[.sh|.bat]</em>).</p>
            <p></p>
          </section>

          <section>
            <title>Running multiprocess file-based example using Tomcat</title>
            <p>In order to run the ManifoldCF multiprocess file-based example under Tomcat, you will need to take the following steps:</p>
            <p></p>
            <ol>
              <li>Start the database (using <em>start-database[.sh|.bat]</em>)</li>
              <li>Initialize the database (using <em>initialize[.sh|.bat]</em>)</li>
              <li>Start the agents process (using <em>start-agents[.sh|.bat]</em>, and optionally <em>start-agents-2[.sh|.bat]</em>)</li>
              <li>Modify the Tomcat startup script, or use the Tomcat service administration client, to set a Java "-Dorg.apache.manifoldcf.configfile" switch to point to the example's <em>properties.xml</em> file.</li>
              <li>Start Tomcat.</li>
              <li>Deploy and start the mcf-crawler-ui, mcf-authority-service, and mcf-api-service web applications, preferably using the Tomcat administration client.</li>
            </ol>
          </section>

        </section>

        <section>
          <title>Simplified multi-process model using ZooKeeper-based synchronization</title>
          <p></p>
          <p>ManifoldCF can be deployed in a simplified multi-process model which uses Apache ZooKeeper to synchronize processes.  Inside the <em>multiprocess-kz-example</em> directory, you will find everything you need to do this.  (The
              <em>multiprocess-zk-example-proprietary</em> directory is similar but includes proprietary material and is available only if you build ManifoldCF yourself.)  Below is a list of
              what you will find in this directory.</p>
          <p></p>
          <table>
            <caption>ZooKeeper-based multiprocess example files and directories</caption>
            <tr><th><em>multiprocess-zk-example</em> file/directory</th><th>Meaning</th></tr>
            <tr><td><em>web</em></td><td>Web applications that should be deployed on tomcat or the equivalent, plus recommended application server -D switch names and values</td></tr>
            <tr><td><em>processes</em></td><td>classpath jars that should be included in the class path for all non-connector-specific processes, along with -D switches, using the same convention as described for tomcat, above</td></tr>
            <tr><td><em>properties.xml</em></td><td>an example ManifoldCF configuration file, in the right place for the multiprocess script to find it</td></tr>
            <tr><td><em>properties-global.xml</em></td><td>an example ManifoldCF shared configuration file, in the right place for the setglobalproperties script to find it</td></tr>
            <tr><td><em>logging.xml</em></td><td>an example ManifoldCF logging configuration file, in the right place for the properties.xml to find it</td></tr>
            <tr><td><em>zookeeper</em></td><td>the example ZooKeeper storage directory, which must be writable in order for ZooKeeper to work</td></tr>
            <tr><td><em>logs</em></td><td>where the ManifoldCF logs get written to</td></tr>
            <tr><td><em>runzookeeper[.sh|.bat]</em></td><td>script to run a ZooKeeper server instance</td></tr>
            <tr><td><em>setglobalproperties[.sh|.bat]</em></td><td>script to initialize ZooKeeper with properties from properties-global.xml</td></tr>
            <tr><td><em>start-database[.sh|.bat]</em></td><td>script to start the HSQLDB database</td></tr>
            <tr><td><em>initialize[.sh|.bat]</em></td><td>script to create the database instance, create all database tables, and register connectors</td></tr>
            <tr><td><em>start-webapps[.sh|.bat]</em></td><td>script to start Jetty with the ManifoldCF web applications deployed</td></tr>
            <tr><td><em>start-agents[.sh|.bat]</em></td><td>script to start (the first) agents process</td></tr>
            <tr><td><em>start-agents-2[.sh|.bat]</em></td><td>script to start a second agents process</td></tr>
            <tr><td><em>stop-agents[.sh|.bat]</em></td><td>script to stop all running agents processes cleanly</td></tr>
          </table>
          <p></p>
          <section>
            <title>Initializing the database and running</title>
            <p></p>
            <p>If you run the ZooKeeper-based multiprocess example, then you must follow the following steps:</p>
            <p></p>
            <ol>
              <li>Start ZooKeeper (using the <em>runzookeeper[.sh|.bat]</em> script)</li>
              <li>Initialize the ManifoldCF shared configuration data (using <em>setglobalproperties[.sh|.bat]</em>)</li>
              <li>Start the database (using <em>start-database[.sh|.bat]</em>)</li>
              <li>Initialize the database (using <em>initialize[.sh|.bat]</em>)</li>
              <li>Start the agents process (using <em>start-agents[.sh|.bat]</em>, and optionally <em>start-agents-2[.sh|.bat]</em>)</li>
              <li>Start the web applications (using <em>start-webapps[.sh|.bat]</em>)</li>
            </ol>
            <p></p>
          </section>

          <section>
            <title>Running multiprocess ZooKeeper example using Tomcat</title>
            <p>In order to run the ManifoldCF ZooKeeper example under Tomcat, you will need to take the following steps:</p>
            <p></p>
            <ol>
              <li>Start ZooKeeper (using the <em>runzookeeper[.sh|.bat]</em> script)</li>
              <li>Initialize the ManifoldCF shared configuration data (using <em>setglobalproperties[.sh|.bat]</em>)</li>
              <li>Start the database (using <em>start-database[.sh|.bat]</em>)</li>
              <li>Initialize the database (using <em>initialize[.sh|.bat]</em>)</li>
              <li>Start the agents process (using <em>start-agents[.sh|.bat]</em>, and optionally <em>start-agents-2[.sh|.bat]</em>)</li>
              <li>Modify the Tomcat startup script, or use the Tomcat service administration client, to set a Java "-Dorg.apache.manifoldcf.configfile" switch to point to the example's <em>properties.xml</em> file.</li>
              <li>Start Tomcat.</li>
              <li>Deploy and start the mcf-crawler-ui, mcf-authority-service, and mcf-api-service web applications, preferably using the Tomcat administration client.</li>
            </ol>
          </section>
          
        </section>

        <section>
          <title>Command-driven multi-process model</title>
          <p></p>
          <p>The most generic way of deploying ManifoldCF involves calling ManifoldCF operations using scripts.  There are a number of java classes among the ManifoldCF classes
            that are intended to be called directly, to perform specific actions in the environment or in the database.  These classes are usually invoked from the command line, with
            appropriate arguments supplied, and are thus considered to be ManifoldCF <strong>commands</strong>.  Basic functionality supplied by these command classes is
            as follows:</p>

          <p></p>
          <ul>
             <li>Create/Destroy the ManifoldCF database instance</li>
             <li>Start/Stop the <strong>agents</strong> process</li>
             <li>Register/Unregister an agent class (there's currently only one included)</li>
             <li>Register/Unregister an output connector</li>
             <li>Register/Unregister a transformation connector</li>
             <li>Register/Unregister a repository connector</li>
             <li>Register/Unregister an authority connector</li>
             <li>Register/Unregister a mapping connector</li>
             <li>Clean up synchronization directory garbage resulting from an ungraceful interruption of an ManifoldCF process</li>
             <li>Query for certain kinds of job-related information</li>
          </ul>
          <p></p>
          <p>Individual connectors may contribute additional command classes and processes to this picture.</p>
          <p></p>
          <p>The multiprocess command execution scripts are delivered in the <em>processes</em> subdirectory.  The script for executing commands is
            <em>processes/executecommand[.sh|.bat]</em>. This script requires two environment variables to be set before execution: JAVA_HOME, and
            MCF_HOME, which should point to ManifoldCF's home execution directory, where the <em>properties.xml</em> file is found.)</p>
            
          <p></p>
          <p>The basic steps required to set up and run ManifoldCF in command-driven file-based multi-process mode are as follows:</p>
          <p></p>
          <ul>
            <li>Install PostgreSQL or MySQL.  The PostgreSQL JDBC driver included with ManifoldCF is 42.x and is thus current through Postgresql 12.  If you want to use MySQL, the ant "download-dependencies" build target will fetch the appropriate MySQL JDBC driver.</li>
            <li>Configure the database for your environment; the default configuration is acceptable for testing and experimentation.</li>
            <li>Create the database instance (see commands below)</li>
            <li>Initialize the database instance (see commands below)</li>
            <li>Register the pull agent (org.apache.manifoldcf.crawler.system.CrawlerAgent, see below)</li>
            <li>Register your connectors and authorities (see below)</li>
            <li>Install a Java application server, such as Tomcat.</li>
            <li>Deploy the war files from <em>web/war</em>, except for <em>mcf-combined.war</em>, to your application server (see below).</li>
            <li>Set the starting environment variables for your app server to include any -D commands found in <em>web/define</em>.  The -D commands should be of the
              form, "-D&#60;file name&#62;=&#60;file contents&#62;".  You will also need a "-Dorg.apache.manifoldcf.configfile=&#60;properties file&#62;" define option, or the
              equivalent, in the application server's JVM startup in order for ManifoldCF to be able to locate its configuration file.</li>
            <li>Use the <em>processes/executecommand[.bat|.sh]</em> command from execute the appropriate commands from the next section below, being sure to first set the
              JAVA_HOME and MCF_HOME environment variables properly.</li>
            <li>Start any supporting processes that result from your build.  (Some connectors such as Documentum and FileNet have auxiliary processes you need to run to make
              these connectors functional.)</li>
            <li>Start your application server.</li>
            <li>Start the ManifoldCF agents process.</li>
            <li>At this point, you should be able to interact with the ManifoldCF UI, which can be accessed via the mcf-crawler-ui web application</li>
          </ul>
          <p></p>
          <p>The detailed list of commands is presented below.</p>
          <p></p>
          <section>
            <title>Commands</title>
            <p></p>
            <p>After you have created the necessary configuration files, you will need to initialize the database, register the "pull-agent" agent, and then register your individual connectors.
              ManifoldCF provides a set of commands for performing these actions, and others as well.  The classes implementing these commands are specified below.</p>
            <p></p>
            <table>
              <tr><th>Core Command Class</th><th>Arguments</th><th>Function</th></tr>
              <tr><td>org.apache.manifoldcf.core.DBCreate</td><td><em>dbuser</em> [<em>dbpassword</em>]</td><td>Create ManifoldCF database instance</td></tr>
              <tr><td>org.apache.manifoldcf.core.DBDrop</td><td><em>dbuser</em> [<em>dbpassword</em>]</td><td>Drop ManifoldCF database instance</td></tr>
              <tr><td>org.apache.manifoldcf.core.LockClean</td><td>None</td><td>Clean out synchronization directory</td></tr>
              <tr><td>org.apache.manifoldcf.core.Obfuscate</td><td>string</td><td>Obfuscate a string, for use as an obfuscated parameter value</td></tr>
            </table>
            <p></p>
            <table>
              <tr><th>Agents Command Class</th><th>Arguments</th><th>Function</th></tr>
              <tr><td>org.apache.manifoldcf.agents.Install</td><td>None</td><td>Create ManifoldCF agents tables</td></tr>
              <tr><td>org.apache.manifoldcf.agents.Uninstall</td><td>None</td><td>Remove ManifoldCF agents tables</td></tr>
              <tr><td>org.apache.manifoldcf.agents.Register</td><td><em>classname</em></td><td>Register an agent class</td></tr>
              <tr><td>org.apache.manifoldcf.agents.UnRegister</td><td><em>classname</em></td><td>Un-register an agent class</td></tr>
              <tr><td>org.apache.manifoldcf.agents.UnRegisterAll</td><td>None</td><td>Un-register all current agent classes</td></tr>
              <tr><td>org.apache.manifoldcf.agents.SynchronizeAll</td><td>None</td><td>Un-register all registered agent classes that can't be found</td></tr>
              <tr><td>org.apache.manifoldcf.agents.RegisterOutput</td><td><em>classname</em> <em>description</em></td><td>Register an output connector class</td></tr>
              <tr><td>org.apache.manifoldcf.agents.UnRegisterOutput</td><td><em>classname</em></td><td>Un-register an output connector class</td></tr>
              <tr><td>org.apache.manifoldcf.agents.UnRegisterAllOutputs</td><td>None</td><td>Un-register all current output connector classes</td></tr>
              <tr><td>org.apache.manifoldcf.agents.SynchronizeOutputs</td><td>None</td><td>Un-register all registered output connector classes that can't be found</td></tr>
              <tr><td>org.apache.manifoldcf.agents.RegisterTransformation</td><td><em>classname</em> <em>description</em></td><td>Register a transformation connector class</td></tr>
              <tr><td>org.apache.manifoldcf.agents.UnRegisterTransformation</td><td><em>classname</em></td><td>Un-register a transformation connector class</td></tr>
              <tr><td>org.apache.manifoldcf.agents.UnRegisterAllTransformations</td><td>None</td><td>Un-register all current transformation connector classes</td></tr>
              <tr><td>org.apache.manifoldcf.agents.SynchronizeTransformations</td><td>None</td><td>Un-register all registered transformation connector classes that can't be found</td></tr>
              <tr><td>org.apache.manifoldcf.agents.AgentRun</td><td>None</td><td>Main <strong>agents</strong> process class</td></tr>
              <tr><td>org.apache.manifoldcf.agents.AgentStop</td><td>None</td><td>Stops the running <strong>agents</strong> process</td></tr>
            </table>
            <p></p>
            <table>
              <tr><th>Crawler Command Class</th><th>Arguments</th><th>Function</th></tr>
              <tr><td>org.apache.manifoldcf.crawler.Register</td><td><em>classname</em> <em>description</em></td><td>Register a repository connector class</td></tr>
              <tr><td>org.apache.manifoldcf.crawler.UnRegister</td><td><em>classname</em></td><td>Un-register a repository connector class</td></tr>
              <tr><td>org.apache.manifoldcf.crawler.UnRegisterAll</td><td>None</td><td>Un-register all repository connector classes</td></tr>
              <tr><td>org.apache.manifoldcf.crawler.SynchronizeConnectors</td><td>None</td><td>Un-register all registered repository connector classes that can't be found</td></tr>
              <tr><td>org.apache.manifoldcf.crawler.ExportConfiguration</td><td><em>filename</em> [<em>passcode</em>]</td><td>Export crawler configuration to a file</td></tr>
              <tr><td>org.apache.manifoldcf.crawler.ImportConfiguration</td><td><em>filename</em> [<em>passcode</em>]</td><td>Import crawler configuration from a file</td></tr>
            </table>
            <p></p>
            <p>NOTE: By adding a passcode as a second argument to the ExportConfiguration command class, the exported file will be encrypted by using the AES algorithm. This can be useful to
              prevent repository passwords to be stored in clear text. In order to use this functionality, you must enter a salt value to your configuration file. The same passcode along
              with the salt value are used to decrypt the file with the ImportConfiguration command class. See the documentation for the commands and properties above to find the
              correct arguments and settings.</p>
            <p></p>
            <table>
              <tr><th>Authorization Domain Command Class</th><th>Arguments</th><th>Function</th></tr>
              <tr><td>org.apache.manifoldcf.authorities.RegisterDomain</td><td><em>domainname</em> <em>description</em></td><td>Register an authorization domain</td></tr>
              <tr><td>org.apache.manifoldcf.authorities.UnRegisterDomain</td><td><em>domainname</em></td><td>Un-register an authorization domain</td></tr>
            </table>
            <p></p>
            <table>
              <tr><th>User Mapping Command Class</th><th>Arguments</th><th>Function</th></tr>
              <tr><td>org.apache.manifoldcf.authorities.RegisterMapper</td><td><em>classname</em> <em>description</em></td><td>Register a mapping connector class</td></tr>
              <tr><td>org.apache.manifoldcf.authorities.UnRegisterMapper</td><td><em>classname</em></td><td>Un-register a mapping connector class</td></tr>
              <tr><td>org.apache.manifoldcf.authorities.UnRegisterAllMappers</td><td>None</td><td>Un-register all mapping connector classes</td></tr>
              <tr><td>org.apache.manifoldcf.authorities.SynchronizeMappers</td><td>None</td><td>Un-register all registered mapping connector classes that can't be found</td></tr>
            </table>
            <p></p>
            <table>
              <tr><th>Authority Command Class</th><th>Arguments</th><th>Function</th></tr>
              <tr><td>org.apache.manifoldcf.authorities.RegisterAuthority</td><td><em>classname</em> <em>description</em></td><td>Register an authority connector class</td></tr>
              <tr><td>org.apache.manifoldcf.authorities.UnRegisterAuthority</td><td><em>classname</em></td><td>Un-register an authority connector class</td></tr>
              <tr><td>org.apache.manifoldcf.authorities.UnRegisterAllAuthorities</td><td>None</td><td>Un-register all authority connector classes</td></tr>
              <tr><td>org.apache.manifoldcf.authorities.SynchronizeAuthorities</td><td>None</td><td>Un-register all registered authority connector classes that can't be found</td></tr>
            </table>
            <p></p>
            <p>Remember that you need to include all the jars under <em>multiprocess-file-example/processes/lib</em> in the classpath whenever you run one of these commands!
                But, luckily, there are scripts which do this for you.  These can be found in <em>multiprocess-file-example/processes/executecommand[.sh,.bat]</em>.
                The scripts require some environment variables to be set, such as <em>MCF_HOME</em> and <em>JAVA_HOME</em>, and expect the configuration file to be
                found at <em>MCF_HOME/properties.xml</em>.</p>
            <p></p>
          </section>
          <section>
            <title>Deploying the <strong>mcf-crawler-ui</strong>, <strong>mcf-authority-service</strong>, and <strong>mcf-api-service</strong> web applications</title>
            <p></p>
            <p>If you built ManifoldCF using ant, then the ant build will have constructed four war files for you under <em>web/war</em>.  You should ignore the <em>mcf-combined</em>
                war in this directory for this deployment model.  If you intend to run ManifoldCF in multiprocess mode, you will need to deploy the other web applications on you application server.
                There is no requirement that the <strong>mcf-crawler-ui</strong>, <strong>mcf-authority-service</strong>, and <strong>mcf-api-service</strong> web
                applications be deployed on the same instance of the application server.  With the current architecture of ManifoldCF, they must be deployed on the same physical server, however.</p>
            <p></p>
            <p>For each of the application servers involved with ManifoldCF, you must set the following define, so that the ManifoldCF web applications can locate the configuration file:</p>
            <source>
-Dorg.apache.manifoldcf.configfile=&#60;configuration file path&#62;
            </source>
            <p></p>
          </section>
          <section>
            <title>Running the <strong>agents</strong> process</title>
            <p></p>
            <p>The <strong>agents</strong> process is the process that actually performs the crawling for ManifoldCF.  Start this process by running the command
              "org.apache.manifoldcf.agents.AgentRun".  This class will run until stopped by invoking the command "org.apache.manifoldcf.agents.AgentStop".  It is highly
              recommended that you stop the process in this way.  You may also stop the process using a SIGTERM signal, but "kill -9" or the equivalent is NOT recommended,
              because that may result in dangling locks in the ManifoldCF synchronization directory.  (If you have to, clean up these locks by shutting down all ManifoldCF
              processes, including the application server instances that are running the web applications, and invoking the command "org.apache.manifoldcf.core.LockClean".)</p>
            <p></p>
          </section>
        </section>
      </section>
      
      <section>
        <title>The <em>connectors.xml</em> configuration file</title>
        <p></p>
        <p>The quick-start, combined, and simplified multi-process sample deployments of ManifoldCF have their own configuration file, called <em>connectors.xml</em>,
          which is used to register the available connectors in the database.  The file has this basic format:</p>
        <p></p>
        <source>
&#60;?xml version="1.0" encoding="UTF-8" ?&#62;
&#60;connectors&#62;
 (clauses)
&#60;/connectors&#62;
        </source>
        <p></p>
        <p>The following tags are available to specify your connectors and authorization domains:</p>
        <p></p>
        <p>&#60;repositoryconnector name="<em>pretty_name</em>" class="<em>connector_class</em>"/&#62;</p>
        <p>&#60;authorityconnector name="<em>pretty_name</em>" class="<em>connector_class</em>"/&#62;</p>
        <p>&#60;mappingconnector name="<em>pretty_name</em>" class="<em>connector_class</em>"/&#62;</p>
        <p>&#60;outputconnector name="<em>pretty_name</em>" class="<em>connector_class</em>"/&#62;</p>
        <p>&#60;transformationconnector name="<em>pretty_name</em>" class="<em>connector_class</em>"/&#62;</p>
        <p>&#60;authorizationdomain name="<em>pretty_name</em>" domain="<em>domain_name</em>"/&#62;</p>
        <p></p>
        <p>The <em>connectors.xml</em> file typically has some connectors commented out - namely the ones build with stubs which require you to supply a
          third-party library in order for the connector to run.  If you build ManifoldCF yourself, the <em>example-proprietary</em> and <em>multiprocess-file-example-proprietary</em>
          and <em>multiprocess-zk-example-proprietary</em> directories instead use <em>connectors-proprietary.xml</em>.  The connectors you build against the proprietary libraries you supply will not have their
          <em>connectors-proprietary.xml</em> tags commented out.</p>
        <p></p>
      </section>

      <section>
        <title>Running connector-specific processes</title>
        <p></p>
        <p>Connector-specific processes require the classpath for their invocation to include all the jars that are in the corresponding
          <em>processes/&#60;process_name&#62;</em> directory.  The Documentum and FileNet connectors are the only two connectors that currently require additional processes. 
          Start these processes using the commands listed below, and stop them with SIGTERM (or ^C, if they are running in a shell).</p>
        <p></p>
        <table>
          <tr><th>Connector</th><th>Process</th><th>Main class</th><th>Script name (relative to <em>dist</em>)</th></tr>
          <tr><td>Documentum</td><td>processes/documentum-server</td><td>org.apache.manifoldcf.crawler.server.DCTM.DCTM</td><td>processes/documentum-server/run[.sh|.bat]</td></tr>
          <tr><td>Documentum</td><td>processes/documentum-registry</td><td>org.apache.manifoldcf.crawler.registry.DCTM.DCTM</td><td>processes/documentum-registry/run[.sh|.bat]</td></tr>
          <tr><td>FileNet</td><td>processes/filenet-server</td><td>org.apache.manifoldcf.crawler.server.filenet.Filenet</td><td>processes/filenet-server/run[.sh|.bat]</td></tr>
          <tr><td>FileNet</td><td>processes/filenet-registry</td><td>org.apache.manifoldcf.crawler.registry.filenet.Filenet</td><td>processes/filenet-registry/run[.sh|.bat]</td></tr>
        </table>
        <p>The registry process in all cases must be started before the corresponding server process, or the server process will report an error.  (It will, however, retry after some period of time.)
            The scripts all require an MCF_HOME environment variable pointing to the place where properties.xml is found, as well as a JAVA_HOME environment variable pointing the JDK.
            The server scripts also require other environment variables as well, consistent with the needs of the DFC or the FileNet API respectively.  For example, DFC requires the
            DOCUMENTUM environment variable to be set, while the FileNet server script requires the WASP_HOME environment variable.</p>
        <p>It is important to understand that the scripts work by building a classpath out of all jars that get copied into the <em>lib</em> and <em>lib-proprietary</em> directory underneath
            each process during the ant build.  The <em>lib-proprietary</em> jars cannot be distributed in the binary version of ManifoldCF, so if you use this option you will still need to
            copy them there yourself for the processes to run.  If you build ManifoldCF yourself, these jars are copied from the <em>lib-proprietary</em> directories underneath the documentum
            or filenet connector directories.  For the server startup scripts to work properly, the <em>lib-proprietary</em> directories should have <strong>all</strong> of the jars needed to
            allow the api code to function.</p>
        <p></p>
      </section>

      <section>
        <title>Database selection</title>
        <p></p>
        <p>You have a variety of open-source databases to choose from when deploying ManifoldCF.  The supported databases each have their own strengths and weaknesses, and
          are listed below:</p>
        <ul>
          <li>PostgreSQL (preferred)</li>
          <li>MySQL (preferred)</li>
          <li>MariaDB (not yet evaluated))</li>
          <li>HSQLDB</li>
        </ul>
        <p>You can select the database of your choice by setting the approprate properties in the applicable <em>properties.xml</em> file.  The choice of database is largely orthogonal
          to the choice of deployment model.  The ManifoldCF deployment examples provided can thus be readily altered to use the database you desire.  The details and caveats of
          each choice is described below.</p>
        <p></p>
        <section>
          <title>Configuring a PostgreSQL database</title>
          <p></p>
          <p>Despite having an internal architecture that cleanly abstracts from specific database details, ManifoldCF is currently fairly specific to PostgreSQL at this time.  There are a number of reasons for this.</p>
          <p></p>
          <ul>
             <li>ManifoldCF uses the database for its document queue, which places a significant load on it.  The back-end database is thus a significant factor in ManifoldCF's performance.  But, in exchange, ManifoldCF benefits enormously from the underlying ACID properties of the database.</li>
             <li>The strategy for getting optimal query plans from the database is not abstracted.  For example, PostgreSQL 8.3+ is very sensitive to certain statistics about a database table, and will not generate a performant plan if the statistics are inaccurate by even a little, in some cases.  So, for PostgreSQL, the database table must be analyzed very frequently, to avoid catastrophically bad plans.  But luckily, PostgreSQL is pretty good at doing analysis quickly.  Oracle, on the other hand, takes a very long time to perform analysis, but its plans are much less sensitive.</li>
             <li>PostgreSQL always does a sequential scan in order to count the number of rows in a table, while other databases return this efficiently.  This has affected the design of the ManifoldCF UI.</li>
             <li>The choice of query form influences the query plan.  Ideally, this is not true, but for both PostgreSQL and for (say) Oracle, it is.</li>
             <li>PostgreSQL has a high degree of parallelism and lack of internal single-threadedness.</li>
          </ul>
          <p></p>
          <p>ManifoldCF has been tested against version 8.3.7, 8.4.5, 9.1, 9.2, 9.3, 10, 11, and 12 of PostgreSQL.  We recommend the following configuration parameter settings to work optimally with ManifoldCF:</p>
          <p></p>
          <ul>
             <li>A default database encoding of UTF-8</li>
             <li><em>postgresql.conf</em> settings as described in the table below</li>
             <li><em>pg_hba.conf</em> settings to allow password access for TCP/IP connections from ManifoldCF</li>
             <li>A maintenance strategy involving cronjob-style vacuuming, rather than PostgreSQL autovacuum</li>
          </ul>
          <p></p>
          <table>
            <caption>Postgresql.conf parameters</caption>
            <tr><th><em>postgresql.conf</em> parameter</th><th>Tested value</th></tr>
            <tr><td>standard_conforming_strings</td><td>on</td></tr>
            <tr><td>shared_buffers</td><td>1024MB</td></tr>
            <tr><td>checkpoint_segments</td><td>300</td></tr>
            <tr><td>maintenance<em>work</em>mem</td><td>2MB</td></tr>
            <tr><td>tcpip_socket</td><td>true</td></tr>
            <tr><td>max_connections</td><td>400</td></tr>
            <tr><td>checkpoint_timeout</td><td>900</td></tr>
            <tr><td>datestyle</td><td>ISO,European</td></tr>
            <tr><td>autovacuum</td><td>off</td></tr>
          </table>
          <p></p>
          <p>Note well: The <em>standard_conforming_strings</em> parameter setting is important to prevent any possibility of SQL injection attacks.  While ManifoldCF
            uses parameterized queries in almost all cases, when it does do string quoting it presumes that the SQL standard for quoting is adhered to.  It is in general good practice
            to set this parameter when working with PostgreSQL for this reason.</p>
          <p></p>
          <section>
            <title>A note about PostgreSQL database maintenance</title>
            <p></p>
            <p>PostgreSQL's architecture causes it to accumulate dead tuples in its data files, which do not interfere with its performance but do bloat the database over time.  The
              usage pattern of ManifoldCF is such that it can cause significant bloat to occur to the underlying PostgreSQL database in only a few days, under sufficient load.  PostgreSQL
              has a feature to address this bloat, called <strong>vacuuming</strong>.  This comes in three varieties: autovacuum, manual vacuum, and manual full vacuum.</p>
            <p></p>
            <p>We have found that PostgreSQL's autovacuum feature is inadequate under such conditions, because it not only fights for database resources pretty much all the time,
              but it falls further and further behind as well.  PostgreSQL's in-place manual vacuum functionality is a bit better, but is still much, much slower than actually making a new
              copy of the database files, which is what happens when a manual full vacuum is performed.</p>
            <p></p>
            <p>Dead-tuple bloat also occurs in indexes in PostgreSQL, so tables that have had a lot of activity may benefit from being reindexed at the time of maintenance. </p>
            <p>We therefore recommend periodic, scheduled maintenance operations instead, consisting of the following:</p>
            <p></p>
            <ul>
             <li>VACUUM FULL VERBOSE;</li>
             <li>REINDEX DATABASE &#60;the_db_name&#62;;</li>
            </ul>
            <p> </p>
            <p>During maintenance, PostgreSQL locks tables one at a time.  Nevertheless, the crawler ui may become unresponsive for some operations, such as when counting
              outstanding documents on the job status page.  ManifoldCF thus has the ability to check for the existence of a file prior to such sensitive operations, and will display a
              useful "maintenance in progress" message if that file is found.  This allows a user to set up a maintenance system that provides adequate feedback for an ManifoldCF
              user of the overall status of the system.</p>
            <p></p>
          </section>
        </section>

        <section>
          <title>Configuring a MySQL database</title>
          <p></p>
          <p>MySQL is not quite as fast as PostgreSQL, but it is a relatively close second in performance tests.  Nevertheless, the ManifoldCF team does not have a large amount
            of experience with this database at this time.  More details will be added to this section as information and experience becomes available.</p>
        </section>
        
        <section>
          <title>Configuring an HSQLDB database</title>
          <p></p>
          <p>HSQLDB's performance seems closely tied to how much of the database can be actually held in memory.  Performance at this time is about half that of PostgreSQL.</p>
          <p>HSQLDB can be used with ManifoldCF in either an embedded fashion (which only works with single-process deployments), or in external fashion, with a database instance running in a separate
            process.  See the <em>properties.xml</em> property descriptions for configuration details.</p>
        </section>
        
      </section>
        
      <section>
        <title>The ManifoldCF configuration files</title>
        <p></p>
        <p>Currently, ManifoldCF requires two configuration files: the main configuration property file, and the logging configuration file.</p>
        <p></p>
        <section>
          <title><em>properties.xml</em> file properties</title>

          <p>The <em>properties.xml</em> property file path can be specified by the system property "org.apache.manifoldcf.configfile".  If not specified through a -D operation, its
              name is presumed to be <em>&#60;user_home&#62;/lcf/properties.xml</em>.  The form of the property file is XML, of the following basic form:</p>
          <p></p>
          <source>
&#60;?xml version="1.0" encoding="UTF-8" ?&#62;
&#60;configuration&#62;
 (clauses)
&#60;/configuration&#62;
          </source>
          <p></p>
          <p>The <em>properties.xml</em> file allows properties to be specified.  A property clause has the form:</p>
          <p></p>
          <p>&#60;property name="<em>property_name</em>" value="<em>property_value</em>"/&#62;</p>
          <p></p>
          <p>One of the optional properties is the name of the logging configuration file.  This property's name is "org.apache.manifoldcf.logconfigfile".  If not present, the logging configuration file will be assumed to be <em>&#60;user_home&#62;/manifoldcf/logging.xml</em>.  The logging configuration file is a standard commons-logging property file, and should be formatted accordingly.</p>
          <p></p>
          <p>Note that all properties described below can also be specified on the command line, via a -D switch.  If both methods of setting the property are used, the -D switch value will override the property file value.</p>
          <p></p>
          <p>The following table describes the configuration property file properties, and what they do:</p>
          <p></p>
          <table>
            <caption>property.xml properties</caption>
            <tr><th>Property</th><th>Required?</th><th>Function</th></tr>
            <tr><td>org.apache.manifoldcf.authimplementationclass</td><td>No</td><td>Auth implementation class (defaults to "org.apache.manifoldcf.core.auth.DefaultAuthenticator", set to "org.apache.manifoldcf.core.auth.LdapAuthenticator" for LDAP)</td></tr>
            <tr><td>org.apache.manifoldcf.login.name</td><td>No</td><td>Default auth implementation crawler UI login user ID (defaults to "admin")</td></tr>
            <tr><td>org.apache.manifoldcf.login.password</td><td>No</td><td>Default auth implementation crawler UI login user password (defaults to "admin")</td></tr>
            <tr><td>org.apache.manifoldcf.login.password.obfuscated</td><td>No</td><td>Default auth implementation obfuscated crawler UI login user password (defaults to "admin")</td></tr>
            <tr><td>org.apache.manifoldcf.apilogin.name</td><td>No</td><td>Default auth implementation API login user ID (defaults to "")</td></tr>
            <tr><td>org.apache.manifoldcf.apilogin.password</td><td>No</td><td>Default auth implementation API login user password (defaults to "")</td></tr>
            <tr><td>org.apache.manifoldcf.apilogin.password.obfuscated</td><td>No</td><td>Default auth implementation obfuscated API login user password (defaults to "")</td></tr>
            <tr><td>org.apache.manifoldcf.login.ldap.providerurl</td><td>No</td><td>LDAP auth implementation provider URL (defaults to "")</td></tr>
            <tr><td>org.apache.manifoldcf.login.ldap.securityauthenticationtype</td><td>No</td><td>LDAP auth implementation auth type (defaults to "simple")</td></tr>
            <tr><td>org.apache.manifoldcf.login.ldap.securityprincipal</td><td>No</td><td>LDAP auth implementation security principal (defaults to "")</td></tr>
            <tr><td>org.apache.manifoldcf.login.ldap.contextsearchquery</td><td>No</td><td>LDAP auth implementation search query (defaults to "")</td></tr>
            <tr><td>org.apache.manifoldcf.login.ldap.searchattribute</td><td>No</td><td>LDAP auth implementation search attribute (defaults to "uid")</td></tr>
            <tr><td>org.apache.manifoldcf.crawleruiwarpath</td><td>Yes, for Jetty</td><td>Location of Crawler UI war</td></tr>
            <tr><td>org.apache.manifoldcf.authorityservicewarpath</td><td>Yes, for Jetty</td><td>Location of Authority Service war</td></tr>
            <tr><td>org.apache.manifoldcf.apiservicewarpath</td><td>Yes, for Jetty</td><td>Location of API Service war</td></tr>
            <tr><td>org.apache.manifoldcf.usejettyparentclassloader</td><td>Yes, for Jetty</td><td>true for single-process example, false for multiprocess example.</td></tr>
            <tr><td>org.apache.manifoldcf.connectorsconfigurationfile</td><td>No</td><td>Location of connectors.xml file, for QuickStart, so ManifoldCF can register connectors.</td></tr>
            <tr><td>org.apache.manifoldcf.dbsuperusername</td><td>No</td><td>Database superuser name, for QuickStart, so ManifoldCF can create database instance.</td></tr>
            <tr><td>org.apache.manifoldcf.dbsuperuserpassword</td><td>No</td><td>Database superuser password, for QuickStart, so ManifoldCF can create database instance.</td></tr>
            <tr><td>org.apache.manifoldcf.dbsuperuserpassword.obfuscated</td><td>No</td><td>Obfuscated database superuser password, for QuickStart, so ManifoldCF can create database instance.</td></tr>
            <tr><td>org.apache.manifoldcf.ui.maxstatuscount</td><td>No</td><td>The maximum number of documents ManifoldCF will try to count for the job status display.  Defaults to 500000.</td></tr>
            <tr><td>org.apache.manifoldcf.databaseimplementationclass</td><td>No</td><td>Specifies the class to use to implement database access.
                Default is a built-in Hsqldb implementation.  Supported choices are:
                org.apache.manifoldcf.core.database.DBInterfacePostgreSQL, 
                org.apache.manifoldcf.core.database.DBInterfaceMySQL,
                org.apache.manifoldcf.core.database.DBInterfaceMariaDB,
                org.apache.manifoldcf.core.database.DBInterfaceHSQLDB</td></tr>
            <tr><td>org.apache.manifoldcf.postgresql.hostname</td><td>No</td><td>PostgreSQL server host name, or localhost if not specified.</td></tr>
            <tr><td>org.apache.manifoldcf.postgresql.port</td><td>No</td><td>PostgreSQL server port, or standard port if not specified.</td></tr>
            <tr><td>org.apache.manifoldcf.postgresql.ssl</td><td>No</td><td>Set to "true" for ssl communication with PostgreSQL.</td></tr>
            <tr><td>org.apache.manifoldcf.mysql.server</td><td>No</td><td>The MySQL or MariaDB server name.  Defaults to 'localhost'.</td></tr>
            <tr><td>org.apache.manifoldcf.mysql.client</td><td>No</td><td>The MySQL or MariaDB client property.  Defaults to 'localhost'.  You may want to set this to '%' for a multi-machine setup.</td></tr>
            <tr><td>org.apache.manifoldcf.mysql.ssl</td><td>No</td><td>Set to "true" for ssl communication with MySQL or MariaDB.</td></tr>
            <tr><td>org.apache.manifoldcf.hsqldbdatabasepath</td><td>No</td><td>Absolute or relative path to HSQLDB database; default is '.'.</td></tr>
            <tr><td>org.apache.manifoldcf.hsqldbdatabaseprotocol</td><td>Yes, for remote HSQLDB connection</td><td>The HSQLDB JDBC protocol; choices are 'hsql', 'http', or 'https'. Default is blank (which means an embedded instance)</td></tr>
            <tr><td>org.apache.manifoldcf.hsqldbdatabaseserver</td><td>Yes, for remote HSQLDB connection</td><td>The HSQLDB remote server name.</td></tr>
            <tr><td>org.apache.manifoldcf.hsqldbdatabaseport</td><td>No</td><td>The HSQLDB remote server port.</td></tr>
            <tr><td>org.apache.manifoldcf.hsqldbdatabaseinstance</td><td>No</td><td>The HSQLDB remote database instance name.</td></tr>
            <tr><td>org.apache.manifoldcf.lockmanagerclass</td><td>No</td><td>Specifies the class to use to implement synchronization.  Default
                is either file-based synchronization or in-memory synchronization, using the org.apache.manifoldcf.core.lockmanager.LockManager class.
                Options include org.apache.manifoldcf.core.lockmanager.BaseLockManager, org.apache.manifoldcf.core.FileLockManager, and
                org.apache.manifoldcf.core.lockmanager.ZooKeeperLockManager.</td></tr>
            <tr><td>org.apache.manifoldcf.synchdirectory</td><td>Yes, if file-based synchronization class is specified</td><td>Specifies the path of a
                synchronization directory.  All ManifoldCF process owners <strong>must</strong> have read/write privileges to this directory.</td></tr>
            <tr><td>org.apache.manifoldcf.zookeeper.connectstring</td><td>Yes, if ZooKeeper-based synchronization class is specified</td><td>Specifies the ZooKeeper
                connection string, consisting of comma-separated hostname:port pairs.</td></tr>
            <tr><td>org.apache.manifoldcf.zookeeper.sessiontimeout</td><td>No</td><td>Specifies the ZooKeeper
                session timeout, if ZooKeeperLockManager is specified.  Defaults to 2000.</td></tr>
            <tr><td>org.apache.manifoldcf.database.maxhandles</td><td>No</td><td>Specifies the maximum number of database connection handles that will by pooled.  Recommended value is 200.</td></tr>
            <tr><td>org.apache.manifoldcf.database.handletimeout</td><td>No</td><td>Specifies the maximum time a handle is to live before it is presumed dead.  Recommend a value of 604800, which is the maximum allowable.</td></tr>
            <tr><td>org.apache.manifoldcf.database.connectiontracking</td><td>No</td><td>True or false.  When "true", will track all allocated database connection handles, and will dump an allocation stack trace when the pool is exhausted.  Useful for diagnosing connection leaks.</td></tr>
            <tr><td>org.apache.manifoldcf.logconfigfile</td><td>No</td><td>Specifies location of logging configuration file.</td></tr>
            <tr><td>org.apache.manifoldcf.database.name</td><td>No</td><td>Describes database name for ManifoldCF; defaults to "dbname" if not specified.</td></tr>
            <tr><td>org.apache.manifoldcf.database.username</td><td>No</td><td>Describes database user name for ManifoldCF; defaults to "manifoldcf" if not specified.</td></tr>
            <tr><td>org.apache.manifoldcf.database.password</td><td>No</td><td>Describes database user's password for ManifoldCF; defaults to "local_pg_password" if not specified.</td></tr>
            <tr><td>org.apache.manifoldcf.database.password.obfuscated</td><td>No</td><td>Obfuscated database user's password for ManifoldCF; defaults to "local_pg_password" if not specified.</td></tr>
            <tr><td>org.apache.manifoldcf.crawler.threads</td><td>No</td><td>Number of crawler worker threads created.  Suggest a value of 30.</td></tr>
            <tr><td>org.apache.manifoldcf.crawler.expirethreads</td><td>No</td><td>Number of crawler expiration threads created.  Suggest a value of 10.</td></tr>
            <tr><td>org.apache.manifoldcf.crawler.cleanupthreads</td><td>No</td><td>Number of crawler cleanup threads created.  Suggest a value of 10.</td></tr>
            <tr><td>org.apache.manifoldcf.crawler.deletethreads</td><td>No</td><td>Number of crawler delete threads created.  Suggest a value of 10.</td></tr>
            <tr><td>org.apache.manifoldcf.crawler.historycleanupinterval</td><td>No</td><td>Milliseconds to retain history records.  Default is 0.  Zero means "forever".</td></tr>
            <tr><td>org.apache.manifoldcf.misc</td><td>No</td><td>Miscellaneous debugging output.  Legal values INFO, WARN, or DEBUG.</td></tr>
            <tr><td>org.apache.manifoldcf.db</td><td>No</td><td>Database debugging output.  Legal values INFO, WARN, or DEBUG.</td></tr>
            <tr><td>org.apache.manifoldcf.lock</td><td>No</td><td>Lock management debugging output.  Legal values INFO, WARN, or DEBUG.</td></tr>
            <tr><td>org.apache.manifoldcf.cache</td><td>No</td><td>Cache management debugging output.  Legal values INFO, WARN, or DEBUG.</td></tr>
            <tr><td>org.apache.manifoldcf.agents</td><td>No</td><td>Agent management debugging output.  Legal values INFO, WARN, or DEBUG.</td></tr>
            <tr><td>org.apache.manifoldcf.perf</td><td>No</td><td>Performance logging debugging output.  Legal values INFO, WARN, or DEBUG.</td></tr>
            <tr><td>org.apache.manifoldcf.crawlerthreads</td><td>No</td><td>Log crawler thread activity.  Legal values INFO, WARN, or DEBUG.</td></tr>
            <tr><td>org.apache.manifoldcf.hopcount</td><td>No</td><td>Log hopcount tracking activity.  Legal values INFO, WARN, or DEBUG.</td></tr>
            <tr><td>org.apache.manifoldcf.jobs</td><td>No</td><td>Log job activity.  Legal values INFO, WARN, or DEBUG.</td></tr>
            <tr><td>org.apache.manifoldcf.connectors</td><td>No</td><td>Log connector activity.  Legal values INFO, WARN, or DEBUG.</td></tr>
            <tr><td>org.apache.manifoldcf.scheduling</td><td>No</td><td>Log document scheduling activity.  Legal values INFO, WARN, or DEBUG.</td></tr>
            <tr><td>org.apache.manifoldcf.authorityconnectors</td><td>No</td><td>Log authority connector activity.  Legal values INFO, WARN, or DEBUG.</td></tr>
            <tr><td>org.apache.manifoldcf.authorityservice</td><td>No</td><td>Log authority service activity.  Legal values are INFO, WARN, or DEBUG.</td></tr>
            <tr><td>org.apache.manifoldcf.salt</td><td>Yes, if file encryption is used</td><td>Specify the salt value to be used for encrypting the file to which the crawler configuration is exported.</td></tr>
          </table>
          <p></p>
          <p>The following table describes 'advanced' configuration property file properties.  They shouldn't need to be changed but provide a greater level of customization:</p>
          <p></p>
          <table>
            <caption>Advanced property.xml properties</caption>
            <tr><th>Property</th><th>Required?</th><th>Default</th><th>Function</th></tr>
            <tr><td>org.apache.manifoldcf.crawler.repository.store_history</td><td>No</td><td>true</td><td>If you do not require reports from within this will disable logging to the repository history (although the reports will still run they will not contain any content). This can increase throughput and reduce the rate of growth of the database.</td></tr>
            <tr><td>org.apache.manifoldcf.db.postgres.analyze.&#60;tablename&#62;</td><td>No</td><td>2000</td><td>For postgresql, specify how many changes should be carried out before carrying out an 'ANALYZE' on the specified table.</td></tr>
            <tr><td>org.apache.manifoldcf.db.postgres.reindex.&#60;tablename&#62;</td><td>No</td><td>250000</td><td>For postgresql, specify how many changes should be carried out before carrying out an 'REINDEX' on the specified table.</td></tr>
            <tr><td>org.apache.manifoldcf.db.mysql.analyze.&#60;tablename&#62;</td><td>No</td><td>2000</td><td>For MySql or MariaDB, specify how many changes should be carried out before carrying out an 'ANALYZE' on the specified table.</td></tr>
            <tr><td>org.apache.manifoldcf.ui.maxstatuscount</td><td>No</td><td>500000</td><td>Set the upper limit for the precise document count to be returned on the 'Status and Job Management' page.</td></tr>
          </table>
          <p></p>
          <p>The configuration file can also specify a set of directories which will be searched for connector jars.  The directive that adds to the class path is:</p>
          <p></p>
          <p>&#60;libdir path="<em>path</em>"/&#62;</p>
          <p></p>
          <p>Note that the path can be relative.  For the purposes of path resolution, "." means the directory in which the <em>properties.xml</em> file is itself located.</p>
          <p></p>
        </section>
          
        <section>
          <title>Logging configuration file properties</title>
          <p></p>
          <p>The <em>logging.xml</em> file contains Apache log4j2 properties in a log4j2 xml format.  The way the
            ManifoldCF logging output is formatted is controlled through this file, as are any loggers that ManifoldCF doesn't explicitly define (e.g. loggers for Apache HttpComponent/HttpClient).
            Other resources are therefore best suited to describe the parameters that can be used and to what effect.</p>
        </section>
          
      </section>
      
      <section>
        <title>Running the ManifoldCF Apache2 plug in</title>
        <p></p>
        <p>The ManifoldCF Apache2 plugin, mod-authz-annotate, is designed to convert an authenticated principle (e.g. from mod-auth-kerb), and query a set of authority services
          for access tokens using an HTTP request.  These access tokens are then passed to a (not included) search engine UI, which can use them to help compose a search that
          properly excludes content that the user is not supposed to see.</p>
        <p></p>
        <p>The list of authority services so queried is configured in Apache's httpd.conf file.  This project includes only one such service: the java authority service, which uses
          authority connections defined in the crawler UI to obtain appropriate access tokens.</p>
        <p></p>
        <p>In order for mod-authz-annotate to be used, it must be placed into Apache2's extensions directory, and configured appropriately in the httpd.conf file.</p>
        <p></p>
        <p>Note: The ManifoldCF project now contains support for converting a Kerberos principal to a list of Active Directory SIDs.  This functionality is contained in the
          Active Directory Authority.  The following connectors are expected to make use of this authority:</p>
        <p></p>
        <ul>
         <li>FileNet</li>
         <li>CIFS</li>
         <li>SharePoint</li>
        </ul>
        <p></p>
        <section>
          <title>Configuring the ManifoldCF Apache2 plug in</title>
          <p></p>
          <p>mod-authz-annotate understands the following httpd.conf commands:</p>
          <p></p>
          <table>
            <tr><th>Command</th><th>Meaning</th><th>Values</th></tr>
            <tr><td>AuthzAnnotateEnable</td><td>Turn on/off the plugin</td><td>"On", "Off"</td></tr>
            <tr><td>AuthzAnnotateAuthority</td><td>Point to an authority service that supports ACL queries, but not ID queries</td><td>The authority URL</td></tr>
            <tr><td>AuthzAnnotateACLAuthority</td><td>Point to an authority service that supports ACL queries, but not ID queries</td><td>The authority URL</td></tr>
            <tr><td>AuthzAnnotateIDAuthority</td><td>Point to an authority service that supports ID queries, but not ACL queries</td><td>The authority URL</td></tr>
            <tr><td>AuthzAnnotateIDACLAuthority</td><td>Point to an authority service that supports both ACL queries and ID queries</td><td>The authority URL</td></tr>
          </table>
        </section>
      </section>
    
      <section>
        <title>Running ManifoldCF with Apache Maven</title>
        <p></p>
        <p>If you build ManifoldCF with Maven, then you will need to run ManifoldCF under Maven.  You currently don't get a lot of options here; the only model
            offered is the QuickStart single process model.  To run it, all you need to do is:</p>
        <source>
cd framework/jetty-runner
mvn exec:exec
        </source>
      </section>

    </section>

    <section>
      <title>Running ManifoldCF tests</title>
      <p></p>
      <p>ManifoldCF has a large suite of tests which helps verify the integrity of a ManifoldCF release.  These tests come in several varieties:</p>
      <p></p>
      <ul>
        <li>Unit tests, which test specific classes</li>
        <li>Integration tests, which test connectors running in the MCF environment</li>
        <li>Load tests, which are designed to run connectors under load over a long period of time</li>
        <li>UI tests, which are like integration tests but are meant to exercise the UI</li>
      </ul>
      <p>There are no special considerations for unit tests.  These are executed across the whole project by the main ant target "run-tests".</p>
      <p>Integration tests have a second dimension of variety, namely which database the test is meant to run on.  The testing databases are:</p>
      <p></p>
      <ul>
        <li>HSQLDB (an embedded database)</li>
        <li>HSQLDBext (running against HSQLDB as an external service)</li>
        <li>PostgreSQL (requires a local installation of PostgreSQL to run against)</li>
        <li>MySQL (requires a local installation of MySQL to run against)</li>
      </ul>
      <p>For both the PostgreSQL and Mysql tests, the databases that are installed locally must have a specific superuser and password set when they are installed, so that the
         tests can create a test database instance and tear it down again.  The superuser name and password for PostgreSQL should be "postgres"/"postgres".  The superuser name
         and password for MySQL should be "root"/"mysql".</p>
      <p>Once set up, you can run the tests you want against the database you want by invoking the ant targets as follows:</p>
      <table>
        <caption>Databases and ant targets</caption>
        <tr><th>Database</th><th>ant targets</th></tr>
        <tr><td>HSQLDB, HSQLDBext</td><td>test-hs, load-hs</td></tr>
        <tr><td>PostgreSQL</td><td>test-pg, load-pg</td></tr>
        <tr><td>MySQL</td><td>test-my, load-my</td></tr>
      </table>
      <p></p>
      <p>Running the UI tests requires some additional software installation and configuration as well.  This is what you will need:</p>
      <p></p>
      <ul>
        <li>Install the Google Chrome browser on your testing machine</li>
        <li>Download the appropriate Selenium Chrome driver from this <a href="https://chromedriver.storage.googleapis.com/index.html?path=2.28/">URL</a></li>
        <li>Unpack the driver executable and place it in the location of your choice</li>
        <li>Set the environment variable <em>SL_CHROME_DRIVER</em> to point to the executable</li>
      </ul>
      <p>After this, you should be able to run the UI tests using the ant target "uitests".  (This executes only the HSQLDB UI tests at this time.)</p>
    </section>
    
    <section>
      <title>Integrating ManifoldCF into another application</title>
      <p></p>
      <p>ManifoldCF can be integrated into another application through a variety of methods.  We'll cover these below.</p>
      <p></p>

      <section>
        <title>Integrating the Quick Start example</title>
        <p></p>
        <p>The Quick Start example can readily be integrated into a single-process application, by using the support methods found in the org.apache.manifoldcf.jettyrunner.ManifoldCFJettyRunner class.
          For the web application components of ManifoldCF, you can either use this class to start them under Jetty, or you can choose to deploy them yourself.  Please note, however, that if you start the ManifoldCF
          agents process within a web application, you are effectively not running a single-process version of ManifoldCF anymore, because each web application will effectively
          have its own set of static classes.</p>
        <p>If you want to try the single-process integration, you should learn what you need by reading the Javadoc for the ManifoldCFJettyRunner class.</p>
      </section>
      
      <section>
        <title>Integrating a multi-process setup</title>
        <p></p>
        <p>In a multi process setup, all of the ManifoldCF processes might as well exist on their own.  You can learn how to programmatically start the agents process by looking at the code
          in the AgentRun command class, as described above.  Similarly, the command classes that register connectors are very small and should be easy to understand.</p>
      </section>

      <section>
          <title>Integrating ManifoldCF with a search engine</title>
          <p></p>
          <p>ManifoldCF's Authority Service is designed to allow maximum flexibility in integrating ManifoldCF security with search engines.  The
                service receives a user identity (as a set of authorization domain/user name tuples), and produces a set of tokens.  It also returns a 
                summary of the status of all authorities that were involved in the assembly of the set of tokens, as a nicety.  A search engine user
                interface could thus signal the user when the results they might be seeing are incomplete, and why.</p>
          <p>The Authority Service expects the following arguments, passed as URL arguments and properly URL encoded:</p>
          <p></p>
          <table>
            <caption>Authority Service URL parameters</caption>
            <tr><th>Authority Service URL parameter</th><th>Meaning</th></tr>
            <tr><td>username</td><td>the username, if there is only one authorization domain</td></tr>
            <tr><td>domain</td><td>the optional authorization domain if there is only one authorization domain (defaults to empty string)</td></tr>
            <tr><td>username_<em>XX</em></td><td>username number <em>XX</em>, where <em>XX</em> is an integer starting at zero</td></tr>
            <tr><td>domain_<em>XX</em></td><td>authorization domain <em>XX</em>, where <em>XX</em> is an integer starting at zero</td></tr>
          </table>
          <p></p>
          <p>Access tokens and authority statuses are returned in the HTTP response separated by newline characters.  Each line has a prefix
                as follows:</p>
          <p></p>
          <table>
            <caption>Authority Service response prefixes</caption>
            <tr><th>Authority Service response prefix</th><th>Meaning</th></tr>
            <tr><td>TOKEN:</td><td>An access token</td></tr>
            <tr><td>AUTHORIZED:</td><td>The name of an authority that found the user to be authorized</td></tr>
            <tr><td>UNREACHABLEAUTHORITY:</td><td>The name of an authority that was found to be unreachable or unusable</td></tr>
            <tr><td>UNAUTHORIZED:</td><td>The name of an authority that found the user to be unauthorized</td></tr>
            <tr><td>USERNOTFOUND:</td><td>The name of an authority that could not find the user</td></tr>
          </table>
          <p></p>
          <p>It is important to remember that only the "TOKEN:" lines actually matter for security.  Even if any of the error conditions apply, the set
                of tokens returned by the Authority Service will be correctly supplied in order to apply appropriate security to documents being searched.</p>
          <p>If you choose to deploy a search-engine plugin supplied by the Apache ManifoldCF project (for example, the Solr plugin), you will not need
                know any of the above, since part of the plugin's purpose is to communicate with the Authority Service and apply the access tokens that are
                returned to the search query automatically.  Some plugins, such as the ElasticSearch plugin, are more or less like toolkits, but still hide most
                of the above from the integrator.  In a more highly customized system, however, you may need to develop your own code which interacts
                with the Authority Service in order to meet your goals.</p>
      </section>

    </section>
  </body>

</document>










