--- loncom/build/readme.html 2001/01/17 12:02:00 1.10 +++ loncom/build/readme.html 2011/03/01 03:43:32 1.24 @@ -1,156 +1,721 @@ -

LON-CAPA Software Developer Instructions

+ + + + + + +LON-CAPA Software Developer Guide + + +

LON-CAPA Software Developer Guide

+

+Created: January 17, 2001 +
Last updated: February 28, 2011 +

+
    +
  1. Using CVS +
  2. +
  3. Viewing the software (make HTML)
  4. +
  5. Compiling the software (make build)
  6. +
  7. Adding/removing files from the LON-CAPA +installation (doc/loncapafiles/loncapafiles.lpml)
  8. +
  9. Updating the LON-CAPA Version number
  10. +
  11. Configurable files versus +non-configurable files
  12. +
  13. Updating the non-configurable files on your +machine (make install)
  14. +
  15. Updating the configurable files on your +machine (make configinstall)
  16. +
  17. Building RPMs (make RPM)
  18. +
-
    -
  1. Using CVS - -
  2. Viewing the software (make HTML) -
  3. Compiling the software (make build) -
  4. Adding/removing files from the LON-CAPA installation (doc/loncapafiles/loncapafiles.html) -
  5. Configurable files versus non-configurable files -
  6. Updating the non-configurable files on your machine (make install) -
  7. Updating the configurable files on your machine (make configinstall) -
  8. Building RPMs (make RPM) -
- -
    - -
  1. Using CVS

    -
    -
  2. -

    Viewing the software (make HTML)

    -Commands - -
    -cd loncom/build
    -rm -Rf HTML (or alternatively, "make clean")
    -make HTML
    -cd HTML
    -(look at the index.html file with a web browser such as Netscape)
    -
    -
    -General description of what happens -

    +

      + +
    1. Using CVS


      +These instructions assume that you are using a Linux or UNIX based +terminal. +
    2. +
    3. +

      Viewing the software (make HTML)

      +

      +Commands +

      +

      +cd loncom/build
      +rm -Rf HTML (or alternatively, "make clean")
      +make HTML
      +cd HTML
      +(look at the index.html file with a web browser such as Netscape) +

      +

      +General description of what happens +

      +

      This is the actual make target code. - -

      +

      +
       
       HTML:
      -        install -d HTML
      -        cp ../../doc/loncapafiles/*.gif HTML
      -        perl parse.pl ../../doc/loncapafiles/loncapafiles.html HTML > HTML/index.html
      +	install -d HTML
      +	cp $(SOURCE)/doc/loncapafiles/*.gif HTML
      +	cat $(SOURCE)/doc/loncapafiles/loncapafiles.lpml | \
      +	perl lpml_parse.pl html development default "$(SOURCE)" '$(TARGET)' \
      +	> HTML/index.html
       
      -
      -
      +
      +

      What basically happens is that specially marked-up data in the LON-CAPA -cvs repository file doc/loncapafiles.html is parsed into a more -viewable format by loncom/build/parse.pl. The resulting -file gives a very well organized view of all the files, directories, +cvs repository file doc/loncapafiles/loncapafiles.lpml is parsed +into a more viewable format by loncom/build/lpml_parse.pl. The +resulting file gives a very well organized view of all the files, directories, links, ownerships, permissions, and brief documentation of what each file does. -

      -
    4. -

      Compiling the software (make build)

      -Commands - -
      +

      +
    5. +
    6. +

      Compiling the software (make build)

      +Commands +

      cd loncom/build -make build - - -General description of what happens -

      +
      make build +

      +

      +General description of what happens +

      +

      This is the actual make target code. - -

      -
      -build:
      -        perl parse.pl ../../doc/loncapafiles/loncapafiles.html build > Makefile.build
      -        make -f Makefile.build all
      -
      -
      - -loncom/build/parse.pl reads in all the build information out -of doc/loncapafiles/loncapafiles.html. A new Makefile named -loncom/build/Makefile.build is dynamically constructed. -This dynamically generated Makefile is then run to build/compile -all the software targets from source. This currently takes 10 minutes -(depends on the speed of the machine you compile with). -

      -Example -

      -Here is information for one file tth.so provided in -doc/loncapafiles/loncapafiles.html. - -

      -
      <METAGROUP> -
      <LONCAPA TYPE=LOCATION DIST="redhat6.2" SOURCE="loncom/modules/TexConvert/tthperl/tth.so" TARGET="usr/lib/perl5/site_perl/5.005/tth.so" CATEGORY="system file"> -
      <DESCRIPTION> -
      shared library file for dynamic loading and unloading of TeX-to-HTML functionality -
      </DESCRIPTION> -
      <BUILD> -
      loncom/modules/TexConvert/tthperl/commands -
      </BUILD> -
      <DEPENDENCIES> -
      ../tthdynamic/tthfunc.c -
      ../ttmdynamic/ttmfunc.c -
      </DEPENDENCIES> -
      - -loncom/build/parse.pl sees the BUILD tags and sets up -a dynamic file Makefile.build to run the command inside the -BUILD tags (currently, DEPENDENCIES is not used for anything -besides documentation). -

      -

      -Here is an example of a dynamically generated Makefile.build -that builds two LON-CAPA files (one of which is tth.so. - -

      +

      +
      +
      +build: Makefile.build pod2html.sh pod2man.sh
      +	echo -n "" > WARNINGS
      +	make -f Makefile.build all
      +	make warningnote
      +
      +Makefile.build: $(SOURCE)/doc/loncapafiles/loncapafiles.lpml lpml_parse.pl
      +	cat $(SOURCE)/doc/loncapafiles/loncapafiles.lpml | \
      +	perl lpml_parse.pl build $(CATEGORY) $(DIST) "$(SOURCE)" "$(TARGET)" \
      +	> Makefile.build
      +
      +
      +

      +loncom/build/lpml_parse.pl reads in all the build information out +of doc/loncapafiles/loncapafiles.lpml. A new Makefile named +loncom/build/Makefile.build is dynamically constructed. +This dynamically generated Makefile is then used to build and compile +all the software targets from source. This can take several minutes +(it depends on the speed of the machine you compile with). +

      +

      +Example +

      +

      +Here is information for one file tth.so provided in +doc/loncapafiles/loncapafiles.lpml. +

      +
      +<file>
      +<source>loncom/homework/caparesponse/capa.so</source>
      +<target dist='default'>usr/lib/perl5/site_perl/5.005/capa.so</target>
      +<target dist='redhat7 redhat7.1'>usr/lib/perl5/site_perl/5.6.0/capa.so</target>
      +<categoryname>system file</categoryname>
      +<description>
      +shared library file for dynamic loading and unloading
      +</description>
      +<build trigger='always run'>
      +loncom/homework/caparesponse/commands
      +</build>
      +<dependencies>
      +caparesponse.c;
      +caparesponse.pm;
      +README;
      +Makefile.PL;
      +capa.i;
      +commands
      +</dependencies>
      +</file>
      +
      +

      +loncom/build/lpml_parse.pl sees the build tags and sets up +a dynamic file Makefile.build to run the command inside the +build tags. The files listed inside the dependencies tags +are included in the Makefile.build so as to determine whether +or not there is a need to compile. +

      +

      +Here is an example of a dynamically generated Makefile.build +that builds two LON-CAPA files (one of which is tth.so). +

      +
       all: ../homework/caparesponse/capa.so ../modules/TexConvert/tthperl/tth.so 
       
      -../homework/caparesponse/capa.so:  ../homework/caparesponse/caparesponse.c ../ho
      -mework/caparesponse/caparesponse.pm alwaysrun
      +../homework/caparesponse/capa.so:  ../homework/caparesponse/caparesponse.c ../homework/caparesponse/caparesponse.pm alwaysrun
               cd ../homework/caparesponse/; sh ./commands
       
      -../modules/TexConvert/tthperl/tth.so:  ../modules/TexConvert/tthperl/../tthdynam
      -ic/tthfunc.c ../modules/TexConvert/tthperl/../ttmdynamic/ttmfunc.c
      +../modules/TexConvert/tthperl/tth.so:  ../modules/TexConvert/tthperl/../tthdynamic/tthfunc.c ../modules/TexConvert/tthperl/../ttmdynamic/ttmfunc.c
               cd ../modules/TexConvert/tthperl/; sh ./commands
       
       alwaysrun:
      -
      -
      -

      -
    7. -

      Adding/removing files from the LON-CAPA installation (doc/loncapafiles/loncapafiles.html)

      -
    8. -

      Configurable files versus non-configurable files

      -
    9. -

      Updating the non-configurable files on your machine (make install)

      -
    10. -

      Updating the configurable files on your machine (make configinstall)

      -
    11. -

      Building RPMs (make RPM)

      -
    + +
  3. +

    Adding/removing files from the LON-CAPA installation + (doc/loncapafiles/loncapafiles.html)

    +

    +To add and remove (and alter) +

    +

    +All that you have to do to alter the behavior of the installation is +edit a single file (doc/loncapafiles/loncapafiles.lpml). +Adding, removing, and altering files requires proper attention +to the syntax of file format of course. +

    +

    +File Format +

    +

    +The preceding "make build" documentation +gives an example of a file entry describing one particular file. +All data within loncapafiles.lpml is specified according +to markup tags. The format and syntax of loncapafiles.lpml +is currently best described by the HTML documentation code at the beginning of +loncapafiles.html (as well as, by example, seeing how various +information is coded). All in all, the syntax is quite simple. +

    +

    +Philosophy and notes (the thing nobody reads) +

    +

    +Packaging the software from CVS onto a machine file system requires many +things: +

    + +

    +I looked into, and tried, different ways of accomplishing the above +including automake and recursive make. The automake system seemed quite +complicated (and needlessly so in terms of this project since, by and large, +it works to coordinate many different types of build/compilation parameters +whereas we are more concerned with installation parameters). The other +alternative, recursive make, +has significant deficiencies since not all the information +is kept in one place, and there are significant levels of dependency +between all the things that must be done to keep software packaging +up to date. A particularly convincing article I found when looking into +much of this was + +"Recursive Make Considered Harmful" by Peter Miller. Other complications +were that, at the time, it was unclear as to what categories +of software files we had, and whether or not the directory structure +of CVS would remain constant. With an ever-developing directory structure +to CVS, I preferred to organize the information on a per-file basis +as opposed to a per-directory basis. +Additionally, a standard big Makefile assumes certain "normalcy" to +the directory structure of different potential operating system directories +(RedHat vs. Debian). +

    +

    +If you take time to look at loncapafiles.lpml +(and perhaps run the make HTML command) +you will find that the organizing information according to the markup +syntax in loncapafiles.lpml is simple. Simple is good. +

    +

    +loncom/build/lpml_parse.pl is the script (invoked automatically +by the various targets in loncom/build/Makefile) that reads +doc/loncapafiles/loncapafiles.lpml. lpml_parse.pl +is capable of reading and returning different types of information +from loncapafiles.lpml depending on how lpml_parse.pl +is invoked. lpml_parse.pl has yet to have introduced new sources +of error, and has been tested in quite a number of ways. As with +any parser however, I remain paranoid. +

    +

    +Finally, some notes on the development. +lpml_parse.pl is very fast and styled after a state-based SAX-like +approach. I do eventually want to use a real XML/XSLT approach, however +I hesitate to make everyone everywhere install something like +XML::Xalan. +Also note that loncapafiles.lpml has a +DTD (loncom/build/lpml.dtd) against which it is valid. +I would also like to use more ENTITY's inside lpml.dtd but currently +the perl XML modules available at CPAN do not digest complex ENTITY's that +well. +

    +

    +The lpml_parse.pl-loncapafiles.lpml +combination has been highly efficient and error-free. +

    +
  4. +
  5. +

    Updating files containing version number on your machine (make aboutVERSION)

    +Commands +

    +cd loncom/build +
    make aboutVERSION +

    +

    +General description of what happens +

    +

    +This is the actual make target code. +

    +
    +
    +aboutVERSION:
    +        cat $(SOURCE)/loncom/license/about.html | \
    +        perl -e '$$d=`date +"%Y%m%d%H"`;chomp($$d);while(<>){s/(\<\!\-\- VERSION \-\-\>|\d+\.\d+\.CVS_HEAD\-\d{10})/$(SHOWVERSION)-$$d/; print;}' > \
    +        $(SOURCE)/loncom/license/about.html.new
    +        mv -v $(SOURCE)/loncom/license/about.html.new \
    +        $(SOURCE)/loncom/license/about.html
    +        cat $(SOURCE)/loncom/loncapa_apache.conf | \
    +        perl -e '$$d=`date +"%Y%m%d%H"`;chomp($$d);while(<>){s/(\<\!\-\- VERSION \-\-\>|\d+\.\d+\.CVS_HEAD\-\d{10})/$(SHOWVERSION)-$$d/; print;}' > \
    +        $(SOURCE)/loncom/loncapa_apache.conf.new
    +        mv -v $(SOURCE)/loncom/loncapa_apache.conf.new \
    +        $(SOURCE)/loncom/loncapa_apache.conf
    +
    +
    +

    +The LON-CAPA Version number is updated in the following files: +

      +
    • loncom/license/about.html
    • +
    • loncom/loncapa_apache.conf
    • +
    +

    +

    +Updating the version information in these files means that visitors to the static HTML page: "About LON-CAPA" on your server will see which version of LON-CAPA you are running. The same information is also displayed on the dynamic log-in page generated by lonlogin.pm. Furthermore, the version of LON-CAPA running on your server can determine which course roles may be selectable on that server, if a course contains resource(s) and/or parameter(s) which require a specific version (or newer) of LON-CAPA. It is also possible for Domain Coordinators in another domain to configure their domain to limit hosting of sessions for users from their own domain to LON-CAPA servers in other domains which are running a specific LON-CAPA version (or newer). +

    +

    On a development server, i.e., one where LON-CAPA is built and installed from a CVS checkout, running make aboutVERSION will cause the version number to be set to something of the form: M.N.CVS_HEAD-YYYYMMDDHH, where M.N will be the version number of the next stable (major) release (e.g., if the current stable release is 3.1; the next release would be 3.2), and YYYYMMDDHH is the datestamp when make aboutVERSION was last run. Note: make aboutVERSION will update the about.html and loncapa_apache.conf in the installation directory tree; it is not until make install is run that files served directly by the web server (or used for configuration) are updated with the versions in that tree. +

    +
  6. +
  7. +

    Configurable files versus non-configurable files

    +

    +Machine-specific information is the difference +

    +

    +The current list of configurable files for the LON-CAPA system is +/etc/httpd/conf/loncapa.conf, +/etc/ntp.conf, +/etc/krb.conf, +/etc/ntp/step-tickers, +/home/httpd/html/res/adm/includes/copyright.tab, +/home/httpd/html/res/adm/includes/un_keyword.tab, +/home/httpd/hosts.tab, and +/home/httpd/spare.tab. +

    +

    +All of these configurable files contain machine-specific information. +For instance, the overall LON-CAPA system relies on unique host IDs such +as msua3, s1, s2, msul1, and 103a1 (specified as a "PerlSetVar lonHostID" +field within /etc/httpd/conf/loncapa.conf). +Non-configurable files simply do NOT have machine-specific information. +

    +

    +The impact on updating software +

    +

    +What this means in terms of software updating is that: +

    +
      +
    • non-configurable files can be simply overwritten with newer versions +(without "anything" else to worry about),
    • +
    • and configurable files must follow these steps to be safely +overwritten: +
        +
      1. have their machine-specific information saved,
      2. +
      3. be overwritten, and then
      4. +
      5. have their machine-specific information restored.
      6. +
      +
    • +
    +
  8. +
  9. +

    Updating the non-configurable files on your machine (make install)

    +Commands +

    +cd loncom/build +
    make install +

    +

    +General description of what happens +

    +

    +This is the actual make target code. +

    +
    +
    +install: TEST_hosts_tab Makefile.install Makefile
    +	echo -n "" > WARNINGS
    +	make -f Makefile.install SOURCE="$(SOURCE)" TARGET="$(TARGET)" \
    +	directories
    +	make -f Makefile.install SOURCE="$(SOURCE)" TARGET="$(TARGET)" files
    +	make -f Makefile.install SOURCE="$(SOURCE)" TARGET="$(TARGET)" links
    +	make SOURCE="$(SOURCE)" TARGET="$(TARGET)" \
    +	NORESTORECONF="$(NORESTORECONF)" configinstall
    +	make postinstall
    +	make warningnote
    +	echo "You can run 'make test' to see if your system is ready to go!"
    +
    +Makefile.install: $(SOURCE)/doc/loncapafiles/loncapafiles.lpml lpml_parse.pl
    +	cat $(SOURCE)/doc/loncapafiles/loncapafiles.lpml | \
    +	perl lpml_parse.pl install $(CATEGORY) $(DIST) "$(SOURCE)" \
    +	"$(TARGET)" > Makefile.install
    +
    +
    +

    +For safety reasons (so as to preserve a machine's configuration), +configuration files are NOT installed during this step. This means +that files such as /etc/httpd/conf/loncapa.conf, +/home/httpd/html/res/adm/includes/copyright.tab, and +/home/httpd/spare.tab are not overwritten, but remain as old, +non-updated copies. (To automatically update these files and save/restore +their encoded machine configuration, you must run "make configinstall"). +

    +
  10. +
  11. +

    Updating the configurable files on your machine (make configinstall)

    +Commands +

    +cd loncom/build +make configinstall +

    +

    +General description of what happens +

    +

    +This is the actual make target code. +

    +
    +
    +configinstall: Makefile.configinstall
    +	make -f Makefile.configinstall SOURCE="$(SOURCE)" TARGET="$(TARGET)" \
    +	configfiles
    +	if (test "0" = $(NORESTORECONF)); then \
    +	perl loncaparestoreconfigurations suffix .lpmlnew; fi
    +
    +Makefile.configinstall: $(SOURCE)/doc/loncapafiles/loncapafiles.lpml lpml_parse.pl
    +	cat $(SOURCE)/doc/loncapafiles/loncapafiles.lpml | \
    +	perl lpml_parse.pl configinstall $(CATEGORY) $(DIST) "$(SOURCE)" \
    +	"$(TARGET)" > Makefile.configinstall
    +
    +
    +

    +Configuration files are installed during this step. This means +that files such as /etc/httpd/conf/loncapa.conf, +/home/httpd/html/res/adm/includes/copyright.tab, and +/home/httpd/spare.tab are overwritten. Before being overwritten, +a backup copy is made though. Information is read out of these +backup copies and restored to the new files by the +loncaparestoreconfigurations script. To ensure that +new file permissions and ownerships are installed, a final set of +chown and chmod commands are called for each of +the configuration files. +

    +

    +For the truly paranoid +

    +

    +If you are truly paranoid, you can just make the +Makefile.configinstall file and then save, copy, +and restore all the configuration values yourself. +loncaparestoreconfigurations is pretty smart though, has yet to +fail, and besides, when needed, backup copies are made. +

    +
  12. +

    Building RPMs (make RPM)

    +

    +LON-CAPA is currently installed through "intelligent tarballs". +What I am describing now is part of an earlier (and perhaps future) effort +involving RPMs. +

    +

    +Commands +

    +

    +cd loncom/build
    +rm -Rf BinaryRoot (or alternatively, "make clean")
    +make RPM
    +(to subsequently install, you can type commands like +"rpm -Uvh --force LON-CAPA-base-3.1-1.i386.rpm") +

    +

    +Configuration files +

    +

    +Configuration files are automatically saved with the file suffix +".rpmsave". So /etc/httpd/conf/loncapa.conf is saved as +/etc/httpd/conf/loncapa.conf.rpmsave. +The loncaparestoreconfigurations script should work to restore +configurations in this case. However, please note that if you install an RPM +twice without restoring your configuration, you will overwrite the +".rpmsave" files. +

    +

    +General description of what happens +

    +

    +This is the actual make target code. +

    +
    +
    +RPM: BinaryRoot base_rpm_file_list
    +	cat $(SOURCE)/doc/loncapafiles/loncapafiles.lpml | \
    +	perl lpml_parse.pl make_rpm $(CATEGORY) $(DIST) $(SOURCE) $(TARGET) \
    +	> base_customizerpm.xml
    +	cat base_rpm_file_list.txt | perl make_rpm.pl base 3.2 1 '' '' \
    +	BinaryRoot base_customizerpm.xml
    +
    +BinaryRoot: base_rpm_file_list
    +	make TARGET='BinaryRoot' NORESTORECONF='1' install
    +
    +base_rpm_file_list:
    +	cat $(SOURCE)/doc/loncapafiles/loncapafiles.lpml | \
    +	perl lpml_parse.pl rpm_file_list $(CATEGORY) $(DIST) $(SOURCE) \
    +	'BinaryRoot' | sort > base_rpm_file_list.txt
    +
    +
    +

    +A BinaryRoot directory is generated that reflects the locations, +ownerships, permissions, and contents for all the CVS source +files, compiled binaries, directories, and links as they should eventually +occur on the '/' filesystem location. +

    +

    +loncom/build/make_rpm.pl (also available at +CPAN) is robust (tested over the +span of months) and, unlike other automated RPM-builders, cleanly +builds new RPMs without any after-effect of temporary files left +on the system. The generated RPM is labeled in the format +LON-CAPA-base-(VERSION)-(RELEASE).i386. VERSION is specified inside the +Makefile. +

    +
  13. +
+ + + + +