--- loncom/build/readme.html 2002/04/27 16:23:40 1.17 +++ loncom/build/readme.html 2011/03/01 03:43:32 1.24 @@ -1,34 +1,34 @@ - + - - + + - -LON-CAPA Software Developer Instructions + +LON-CAPA Software Developer Guide -

LON-CAPA Software Developer Instructions

+

LON-CAPA Software Developer Guide

-
Written by Scott Harrison, January 17, 2001 -
Last updated, April 27, 2002 +Created: January 17, 2001 +
Last updated: February 28, 2011

    -
  1. Using CVS
  2. +
  3. Using CVS +
  4. Viewing the software (make HTML)
  5. Compiling the software (make build)
  6. Adding/removing files from the LON-CAPA installation (doc/loncapafiles/loncapafiles.lpml)
  7. +
  8. Updating the LON-CAPA Version number
  9. Configurable files versus non-configurable files
  10. Updating the non-configurable files on your @@ -39,50 +39,42 @@ machine (make configinstall)
    - -
  1. Using CVS

  2. + +
  3. Using CVS


    These instructions assume that you are using a Linux or UNIX based terminal.
    +
  4. -

    Viewing the software (make HTML)

  5. +

    Viewing the software (make HTML)

    +

    Commands - -

    -cd loncom/build
    -rm -Rf HTML (or alternatively, "make clean")
    -make HTML
    -cd HTML
    -(look at the index.html file with a web browser such as Netscape)
    -
    - +

    +

    +cd loncom/build
    +rm -Rf HTML (or alternatively, "make clean")
    +make HTML
    +cd HTML
    +(look at the index.html file with a web browser such as Netscape) +

    +

    General description of what happens +

    This is the actual make target code. - +

     
     HTML:
    @@ -260,29 +264,28 @@ HTML:
     	> HTML/index.html
     
     
    -
    +

    What basically happens is that specially marked-up data in the LON-CAPA -cvs repository file doc/loncapafiles.lpml is parsed into a more -viewable format by loncom/build/lpml_parse.pl. The resulting -file gives a very well organized view of all the files, directories, +cvs repository file doc/loncapafiles/loncapafiles.lpml is parsed +into a more viewable format by loncom/build/lpml_parse.pl. The +resulting file gives a very well organized view of all the files, directories, links, ownerships, permissions, and brief documentation of what each file does.

    +
  6. Compiling the software (make build)

    Commands - -
    +

    cd loncom/build -make build -

    - +
    make build +

    General description of what happens

    This is the actual make target code. - +

     
     build: Makefile.build pod2html.sh pod2man.sh
    @@ -296,19 +299,21 @@ Makefile.build: $(SOURCE)/doc/loncapafil
     	> Makefile.build
     
     
    -
    +

    loncom/build/lpml_parse.pl reads in all the build information out of doc/loncapafiles/loncapafiles.lpml. A new Makefile named loncom/build/Makefile.build is dynamically constructed. -This dynamically generated Makefile is then run to build/compile -all the software targets from source. This currently takes 10 minutes -(depends on the speed of the machine you compile with). +This dynamically generated Makefile is then used to build and compile +all the software targets from source. This can take several minutes +(it depends on the speed of the machine you compile with).

    +

    Example +

    Here is information for one file tth.so provided in doc/loncapafiles/loncapafiles.lpml. - +

     <file>
     <source>loncom/homework/caparesponse/capa.so</source>
    @@ -331,7 +336,7 @@ commands
     </dependencies>
     </file>
     
    -
    +

    loncom/build/lpml_parse.pl sees the build tags and sets up a dynamic file Makefile.build to run the command inside the build tags. The files listed inside the dependencies tags @@ -341,7 +346,7 @@ or not there is a need to compile.

    Here is an example of a dynamically generated Makefile.build that builds two LON-CAPA files (one of which is tth.so). - +

     all: ../homework/caparesponse/capa.so ../modules/TexConvert/tthperl/tth.so 
     
    @@ -353,20 +358,23 @@ all: ../homework/caparesponse/capa.so ..
     
     alwaysrun:
     
    -
    -

  7. -

    Adding/removing files from the LON-CAPA installation (doc/loncapafiles/loncapafiles.html)

    +

    Adding/removing files from the LON-CAPA installation + (doc/loncapafiles/loncapafiles.html)

    +

    To add and remove (and alter) +

    All that you have to do to alter the behavior of the installation is edit a single file (doc/loncapafiles/loncapafiles.lpml). Adding, removing, and altering files requires proper attention to the syntax of file format of course.

    +

    File Format -

    -The preceding "make build" documentation +

    +

    +The preceding "make build" documentation gives an example of a file entry describing one particular file. All data within loncapafiles.lpml is specified according to markup tags. The format and syntax of loncapafiles.lpml @@ -374,10 +382,13 @@ is currently best described by the HTML loncapafiles.html (as well as, by example, seeing how various information is coded). All in all, the syntax is quite simple.

    +

    Philosophy and notes (the thing nobody reads) +

    Packaging the software from CVS onto a machine file system requires many things: +

    -

    I looked into, and tried, different ways of accomplishing the above including automake and recursive make. The automake system seemed quite complicated (and needlessly so in terms of this project since, by and large, it works to coordinate many different types of build/compilation parameters -whereas we are more concerned with installation parameters). Recursive make -has significant deficiencies in the sense that not all the information +whereas we are more concerned with installation parameters). The other +alternative, recursive make, +has significant deficiencies since not all the information is kept in one place, and there are significant levels of dependency between all the things that must be done to keep software packaging up to date. A particularly convincing article I found when looking into much of this was -"Recursive Make Considered Harmful" by Peter Miller. Complicating -matters was, at the time, it was unclear as to what categories +"Recursive Make Considered Harmful" by Peter Miller. Other complications +were that, at the time, it was unclear as to what categories of software files we had, and whether or not the directory structure of CVS would remain constant. With an ever-developing directory structure to CVS, I preferred to organize the information on a per-file basis -as opposed to a per-directory basis (although there is a successful -implementation of a standard big Makefile in loncom/Makefile). +as opposed to a per-directory basis. Additionally, a standard big Makefile assumes certain "normalcy" to the directory structure of different potential operating system directories (RedHat vs. Debian).

    -If you take time to look at loncapafiles.lpml +If you take time to look at loncapafiles.lpml (and perhaps run the make HTML command) you will find that the organizing information according to the markup syntax in loncapafiles.lpml is simple. Simple is good. @@ -443,66 +453,118 @@ of error, and has been tested in quite a any parser however, I remain paranoid.

    +Finally, some notes on the development. lpml_parse.pl is very fast and styled after a state-based SAX-like -approach. Additionally, loncapafiles.lpml has a -DTD (loncom/build/lpml.dtd) against which it is valid. -I would like to use more ENTITY's inside lpml.dtd but currently -Perl XML modules available at CPAN do not digest complex ENTITY's that well. +approach. I do eventually want to use a real XML/XSLT approach, however +I hesitate to make everyone everywhere install something like +XML::Xalan. +Also note that loncapafiles.lpml has a +DTD (loncom/build/lpml.dtd) against which it is valid. +I would also like to use more ENTITY's inside lpml.dtd but currently +the perl XML modules available at CPAN do not digest complex ENTITY's that +well.

    The lpml_parse.pl-loncapafiles.lpml -combination has been working very efficiently and error-free. +combination has been highly efficient and error-free. +

    +
  8. +
  9. +

    Updating files containing version number on your machine (make aboutVERSION)

    +Commands +

    +cd loncom/build +
    make aboutVERSION +

    +

    +General description of what happens +

    +

    +This is the actual make target code.

    -
  10. +
    +
    +aboutVERSION:
    +        cat $(SOURCE)/loncom/license/about.html | \
    +        perl -e '$$d=`date +"%Y%m%d%H"`;chomp($$d);while(<>){s/(\<\!\-\- VERSION \-\-\>|\d+\.\d+\.CVS_HEAD\-\d{10})/$(SHOWVERSION)-$$d/; print;}' > \
    +        $(SOURCE)/loncom/license/about.html.new
    +        mv -v $(SOURCE)/loncom/license/about.html.new \
    +        $(SOURCE)/loncom/license/about.html
    +        cat $(SOURCE)/loncom/loncapa_apache.conf | \
    +        perl -e '$$d=`date +"%Y%m%d%H"`;chomp($$d);while(<>){s/(\<\!\-\- VERSION \-\-\>|\d+\.\d+\.CVS_HEAD\-\d{10})/$(SHOWVERSION)-$$d/; print;}' > \
    +        $(SOURCE)/loncom/loncapa_apache.conf.new
    +        mv -v $(SOURCE)/loncom/loncapa_apache.conf.new \
    +        $(SOURCE)/loncom/loncapa_apache.conf
    +
    +
    +

    +The LON-CAPA Version number is updated in the following files: +

      +
    • loncom/license/about.html
    • +
    • loncom/loncapa_apache.conf
    • +
    +

    +

    +Updating the version information in these files means that visitors to the static HTML page: "About LON-CAPA" on your server will see which version of LON-CAPA you are running. The same information is also displayed on the dynamic log-in page generated by lonlogin.pm. Furthermore, the version of LON-CAPA running on your server can determine which course roles may be selectable on that server, if a course contains resource(s) and/or parameter(s) which require a specific version (or newer) of LON-CAPA. It is also possible for Domain Coordinators in another domain to configure their domain to limit hosting of sessions for users from their own domain to LON-CAPA servers in other domains which are running a specific LON-CAPA version (or newer). +

    +

    On a development server, i.e., one where LON-CAPA is built and installed from a CVS checkout, running make aboutVERSION will cause the version number to be set to something of the form: M.N.CVS_HEAD-YYYYMMDDHH, where M.N will be the version number of the next stable (major) release (e.g., if the current stable release is 3.1; the next release would be 3.2), and YYYYMMDDHH is the datestamp when make aboutVERSION was last run. Note: make aboutVERSION will update the about.html and loncapa_apache.conf in the installation directory tree; it is not until make install is run that files served directly by the web server (or used for configuration) are updated with the versions in that tree. +

    +
  11. +
  12. Configurable files versus non-configurable files

    Machine-specific information is the difference

    The current list of configurable files for the LON-CAPA system is -/etc/httpd/access.conf, /etc/smb.conf, /etc/ntp.conf, /etc/krb.conf, -/etc/atalk/config, /etc/ntp/step-tickers, -/home/httpd/html/res/adm/includes/copyright.tab, -/home/httpd/html/res/adm/includes/un_keyword.tab, -/home/httpd/hosts.tab, and -/home/httpd/spare.tab. +/etc/httpd/conf/loncapa.conf, +/etc/ntp.conf, +/etc/krb.conf, +/etc/ntp/step-tickers, +/home/httpd/html/res/adm/includes/copyright.tab, +/home/httpd/html/res/adm/includes/un_keyword.tab, +/home/httpd/hosts.tab, and +/home/httpd/spare.tab.

    All of these configurable files contain machine-specific information. -For instance, the LON-CAPA system relies on unique host IDs such +For instance, the overall LON-CAPA system relies on unique host IDs such as msua3, s1, s2, msul1, and 103a1 (specified as a "PerlSetVar lonHostID" -field within /etc/httpd/access.conf). +field within /etc/httpd/conf/loncapa.conf). Non-configurable files simply do NOT have machine-specific information. +

    +

    The impact on updating software +

    -What this means in terms of software updating is that +What this means in terms of software updating is that: +

    • non-configurable files can be simply overwritten with newer versions (without "anything" else to worry about),
    • and configurable files must follow these steps to be safely -overwritten
    • +overwritten:
        -
      1. have their machine specific information saved,
      2. +
      3. have their machine-specific information saved,
      4. be overwritten, and then
      5. -
      6. have their machine specific information restored.
      7. +
      8. have their machine-specific information restored.
      +
    -

    +
  13. Updating the non-configurable files on your machine (make install)

    Commands - -
    +

    cd loncom/build -make install -

    - +
    make install +

    General description of what happens

    This is the actual make target code. - +

     
     install: TEST_hosts_tab Makefile.install Makefile
    @@ -523,43 +585,48 @@ Makefile.install: $(SOURCE)/doc/loncapaf
     	"$(TARGET)" > Makefile.install
     
     
    -
    -For safety reasons (so as to not mess up a machine's configuration), +

    +For safety reasons (so as to preserve a machine's configuration), configuration files are NOT installed during this step. This means -that files such as /etc/httpd/access.conf, /etc/smb.conf, /etc/atalk/config, -/home/httpd/html/res/adm/includes/copyright.tab, and -/home/httpd/spare.tab are not overwritten, but remain as old, non-updated -copies. (To automatically update these files and save/restore +that files such as /etc/httpd/conf/loncapa.conf, +/home/httpd/html/res/adm/includes/copyright.tab, and +/home/httpd/spare.tab are not overwritten, but remain as old, +non-updated copies. (To automatically update these files and save/restore their encoded machine configuration, you must run "make configinstall").

    +
  14. Updating the configurable files on your machine (make configinstall)

    Commands - -
    +

    cd loncom/build make configinstall -

    - +

    +

    General description of what happens +

    This is the actual make target code. - +

     
    -configinstall: 
    -        # there is a dependency on having directories in place, but oh well...
    -        perl parse.pl ../../doc/loncapafiles/loncapafiles.html configinstall > Makefile.configinstall
    -        make -f Makefile.configinstall SOURCE="../.." TARGET="" configfiles
    -        perl loncaparestoreconfigurations lasttimestamp
    -        make -f Makefile.configinstall TARGET="" configpermissions
    +configinstall: Makefile.configinstall
    +	make -f Makefile.configinstall SOURCE="$(SOURCE)" TARGET="$(TARGET)" \
    +	configfiles
    +	if (test "0" = $(NORESTORECONF)); then \
    +	perl loncaparestoreconfigurations suffix .lpmlnew; fi
    +
    +Makefile.configinstall: $(SOURCE)/doc/loncapafiles/loncapafiles.lpml lpml_parse.pl
    +	cat $(SOURCE)/doc/loncapafiles/loncapafiles.lpml | \
    +	perl lpml_parse.pl configinstall $(CATEGORY) $(DIST) "$(SOURCE)" \
    +	"$(TARGET)" > Makefile.configinstall
     
     
    -
    +

    Configuration files are installed during this step. This means -that files such as /etc/httpd/access.conf, /etc/smb.conf, /etc/atalk/config, -/home/httpd/html/res/adm/includes/copyright.tab, and -/home/httpd/spare.tab are overwritten. Before being overwritten, +that files such as /etc/httpd/conf/loncapa.conf, +/home/httpd/html/res/adm/includes/copyright.tab, and +/home/httpd/spare.tab are overwritten. Before being overwritten, a backup copy is made though. Information is read out of these backup copies and restored to the new files by the loncaparestoreconfigurations script. To ensure that @@ -575,39 +642,35 @@ If you are truly paranoid, you can just Makefile.configinstall file and then save, copy, and restore all the configuration values yourself. loncaparestoreconfigurations is pretty smart though, has yet to -fail, and besides, when needed backup copies are made. +fail, and besides, when needed, backup copies are made.

    -
  15. +
  16. Building RPMs (make RPM)

    -Commands - -
    -cd loncom/build
    -rm -Rf BinaryRoot (or alternatively, "make clean")
    -make RPM
    -   (to subsequently install, you can type commands like
    -       "rpm -Uvh --force LON-CAPA-base-3.1-1.i386.rpm")
    -
    -
    +

    +LON-CAPA is currently installed through "intelligent tarballs". +What I am describing now is part of an earlier (and perhaps future) effort +involving RPMs.

    -WARNING!!!!!!!!!!!!!!

    -Never never never never never manually install the -LON-CAPA-setup-3.1-1.i386.rpm. This RPM is meant to only be -installed by the CD installation process (it wipes out -the existing /etc/passwd file). +Commands

    +

    +cd loncom/build
    +rm -Rf BinaryRoot (or alternatively, "make clean")
    +make RPM
    +(to subsequently install, you can type commands like +"rpm -Uvh --force LON-CAPA-base-3.1-1.i386.rpm") +

    Configuration files

    Configuration files are automatically saved with the file suffix -".rpmsave". So /etc/httpd/conf/access.conf is saved as -/etc/httpd/conf/access.conf.rpmsave. You can restore -the machine-specific configuration information by running -the /usr/sbin/loncaparestoreconfigurations. However, -a warning is important here. If you install an RPM twice -without restoring your configuration, you will overwrite the +".rpmsave". So /etc/httpd/conf/loncapa.conf is saved as +/etc/httpd/conf/loncapa.conf.rpmsave. +The loncaparestoreconfigurations script should work to restore +configurations in this case. However, please note that if you install an RPM +twice without restoring your configuration, you will overwrite the ".rpmsave" files.

    @@ -615,14 +678,14 @@ without restoring your configuration, yo

    This is the actual make target code. - +

     
     RPM: BinaryRoot base_rpm_file_list
     	cat $(SOURCE)/doc/loncapafiles/loncapafiles.lpml | \
     	perl lpml_parse.pl make_rpm $(CATEGORY) $(DIST) $(SOURCE) $(TARGET) \
     	> base_customizerpm.xml
    -	cat base_rpm_file_list.txt | perl make_rpm.pl base 3.2 '' '' \
    +	cat base_rpm_file_list.txt | perl make_rpm.pl base 3.2 1 '' '' \
     	BinaryRoot base_customizerpm.xml
     
     BinaryRoot: base_rpm_file_list
    @@ -634,26 +697,25 @@ base_rpm_file_list:
     	'BinaryRoot' | sort > base_rpm_file_list.txt
     
     
    -
    +

    A BinaryRoot directory is generated that reflects the locations, ownerships, permissions, and contents for all the CVS source files, compiled binaries, directories, and links as they should eventually occur on the '/' filesystem location.

    -loncom/build/make_rpm.pl is robust (tested over the +loncom/build/make_rpm.pl (also available at +CPAN) is robust (tested over the span of months) and, unlike other automated RPM-builders, cleanly builds new RPMs without any after-effect of temporary files left -on the system. (On the negative side, there are a number of -LON-CAPA specific customizations inside make_rpm.pl which, for -the sake of reusability, should eventually be removed). Two new RPMs -are generated: LON-CAPA-base-3.1-1.i386 and LON-CAPA-setup-3.1-1.i386.rpm -(again, never manually install LON-CAPA-setup-3.1-1.i386.rpm). +on the system. The generated RPM is labeled in the format +LON-CAPA-base-(VERSION)-(RELEASE).i386. VERSION is specified inside the +Makefile.

    +
-