Functional Specification: Application Server Administration

Table of Contents (Status: Draft, Ready for Review, Reviewed, ARC'ed )
1. Introduction

1.1 Project/Component Working Name
1.2 Name(s) and e-mail address of Document Author(s)/Supplier
1.3. Date of This Document
2. Project Summary
2.1 Project Description
2.2 Risks and Assumptions
3. Problem Summary
3.1 Problem Area
3.2 Justification
4. Technical Description

4.1 Details
4.2 Bugs/RFE's
4.3 Scope
4.4 Out-of-scope
4.5 Interfaces
4.6 Documentation Impact
4.7 Configuration/administration Impact
4.8 High Availability Impact
4.9 Internationalization
4.10 Packaging
4.11 Security Impact
4.12 Compatibility
4.12 Dependencies

5. References
6. Schedule
7. Document History


1. Introduction

1.1. Project/Component Working Name

GlassFish V2/Sun Java System Application Server 9.1.

1.2. Name(s) and e-mail address of Document Author(s)/Supplier

Kedar Mhaswade (Kedar.Mhaswade@Sun.Com)

1.3. Date of This Document

09/01/2006.

2. Project Summary


2.1. Project Description

The project is about administration and management support for GlassFish V2. This also covers the overall domain configuration. It includes the following areas: Domain Configuration Management, Administrative MBeans and JMX Infrastructure, Inter Process Communication using JMX Connectors, Management API, JSR 77 Support, Administrative Security, Process Launcher and Startup, Synchronization, Dynamic Reconfiguration, and Node Agent.

It is assumed that the reader is familiar with the overall application server architecture. The general administration architecture of application server is more or less similar to that of the previous releases. The GlassFish V2 (Application server 9.1) administrative projects are focused more on the improved scalability, performance and ease-of-use of existing feature set.

Since most of the server side entities are being enhanced in this release, this document just lists the enhancements/changes made to the existing features, along with new features, if any.

2.2. Risks and Assumptions

There are no risks in this project.

3. Problem Summary

3.1. Problem Area

The administration experience of application server should be enhanced. It is also important that the scalability and performance of the Domain Admin Server is improved.

3.2. Justification

Administration of application server should always be improved and made more robust.

4. Technical Description

4.1. Details

Here is a list of things that we will be doing, for this release.

Item Details
Domain Configuration Management

Background

The configuration is stored in various files on disk. There are no new configuration files planned for this release.

Changes planned for this release

  • The domain's configuration DTD changes are being tracked independently at: http://www.glassfishwiki.org/gfwiki/Wiki.jsp?page=GlassFishV2ApplicationServer9.1
  • Administrative interface will be provided to default web tier configuration (that is stored in default-web.xml). The exact list of settings that will be managed is TBD.
    • The default web-tier configuration is applicable to all the web applications deployed in the domain.
    • All the affected servers will require a restart. In other words, changes made to default-web.xml are not dynamic.

    The default-web.xml editing support is NOT PLANNED for this release.

  • It will be possible to copy the entire domain configuration folder (e.g. <install-dir>/domains/domain1) somewhere else on the same host. The idea is that a domain's configuration does not depend on where the application software bits are stored on the disk. It is the user/administrator responsibility to make sure that the source and target versions of the application server are the same. Following should be noted in this regard:
    • This feature should not be confused with backup-restore of the domain configuration. One is not a replacement of the other.
    • The start-domain command might have to change, after the domain configuration has moved from one place to another.
    • The name of the domain (root folder containing domain's configuration) must remain the same.
  • It will also be possible to copy the entire installation somewhere else on the given host (6418805, 90). Following should be noted in this regard:
    • All the scripts in the <install-dir>/bin folder should continue to work seamlessly, as long as their "execute" permission is retained.
Administrative MBeans and JMX Infrastructure

Background

These are the MBeans that pertain to the configuration changes of the domain. The Admin GUI and CLI usually communicate with these MBeans to perform configuration tasks.

Changes planned for this release

No changes are planned here.

Inter Process Communication using JMX Connectors

Background

The JMX Connectors are used for two distinct purposes within application server:

  • One-way communication between asadmin client and DAS. This is a proprietary implementation of client-side JSR-160, over HTTP/HTTPS.
  • Two-way communication between DAS and node-agents + other server instances. DAS communicates with these over RMI/JMX -- the standard JSR-160 Connector that is integrated into the Java Platform.

Changes planned for this release

No changes are planned here, apart from some bug-fixes.

Management API (AMX)

Background

AMX is the programmatic API to manage application server.

Changes planned for this release

  • Provide the offline configuration support for most of the configuration that is stored in the domain.xml. This means that when a user wants to modify domain.xml while the DAS is not running, s/he should be able to do that. This is required when the user knows the configuration changes to be made and does not want to start the DAS to be running. Till now, in order to modify the domain's configuration, we need the DAS to be running. Following should be considered in this regard:
    • It is not required that application archive deployment is supported with this feature. The reason is that the deployment is quite complex operation and is not supported outside the running DAS In fact, following operations are not required to be supported using offline configuration, for this release:
      • Deployment of application archives.
      • Creation of users/groups that use the FileRealm.
      • Load Balancer configuration.
    • Minimum validation of the domain's configuration must be done.
    • It should be ensured by AMX that domain is NOT running, while someone tries to perform offline configuration.
    • No Dynamic reconfiguration support will be provided. In other words, the affected server instances will have to be restarted so that their configuration is modified. In general, administrators should make sure that no server side entities are running when offline configuration is performed.
    • This facility is meant mainly for the configurators that want to modify the domain's configuration in a certain manner, without having to start the DAS
    Offline Configuration is NOT PLANNED for this release
  • Provide a way to reload an application.
JSR 77 Support

Background

JSR 77 Specification is in maintenance mode, so there is no new revision of the specification.

Changes planned for this release

  • The first time startup of the domain/instance causes the MEjb application (which is just an EJB application) to be deployed. This causes some delay in bringing up the domain/instance. It also incurs minor overhead to make sure that it is not required to be deployed, on every startup. We are going to prepackage the MEjb application, so that the startup time is improved. This will marginally increase the application server bundle size.
  • See the administrative security section for other changes made to MEjb access by the JSR 77 clients.
Administrative Security

Background

This section covers the changes being made to the administrative security of application server.

Changes planned for this release

There are no major changes planned for administrative security.

  • Currently, a group-name, asadmin is reserved for the group of administrators -- the users of application server, who have administrative privileges. This works fine when the default authentication realm (file-realm) is used. But when, LDAP is used for the authentication, unfortunately all the users in this group get administrative access to all the domains! This is a customer issue 6454224 which will be fixed for this release.
  • We should be able to use either JKS or NSS as our key-store and/or trust-store. Currently, by default, the Platform Edition of the product gets JKS, where as Enterprise Edition gets NSS. For this release, rather than hard-coding these, there will be a way to choose what store is needed by an administrator for a particular domain. Note however that, once a domain has a particular kind of store, it will not be possible to change it. These details are covered in the one pager for usage profiles.
Process Launcher and Startup/Shutdown

Background

The process launcher code is responsible for launching the application server process. In the default mode, a separate launcher VM is started that finally starts the application server JVM (the launched JVM). This is the default for this release. In case of the other flavor, a native process is started (the launcher process) that launches another native application server process which embeds a JVM. We will need to maintain both the flavors for this release.

Changes planned for this release

  • Improve the startup performance in the case of default mode: Currently, when the Java Launcher is used, there is a Launcher JVM (2) that is started and that invokes the application server JVM (3).
             ______________         ______________       _______________
            | asadmin JVM  | ----- | Launcher JVM |-----| App Server JVM|
            |______(1)_____|       |_____(2)______|     |_______(3)_____|
                    |                                            |
                    |__(proposed) asadmin JVM -> app server______|
            

It is desired that we eliminate the separate Launcher JVM and make the asadmin JVM itself a launching JVM. For that matter, it should be possible for any JVM to launch the app server JVM as a separate process. This will be made part of the AMX so that it becomes a public API. Note that following in this regard:

  • If possible, streaming API for XML will be used to parse the domain.xml faster, to form the Java invocation command line.
  • There are some hairy issues here from a compatibility standpoint. For example, it is imperative that all the options on the start-domain and start-instance commands are taken care of, while we try to make this change.
  • The changes should apply to both start-domain and start-instance commands, in the case of start-instance command, the node-agent being the launching VM.
  • The Java-level thread-dump capability of the server must be preserved. Although we have several ways to get the thread dump from the app server, the only reliable way is to send an OS signal (e.g. SIGQUIT on Solaris) to the hung JVM. Something that must be noted (which is actually a Java behavior) is that by default, while using the Java Launcher, we cannot get the thread dump in the application server's server.log, by sending the SIGQUIT-like signal to the JVM process. This is because the signal handler in the JVM (native code) does not recognize the PrintStream that Java-land understands.
  • There used to be an internal interface called processlauncher.xml. The stability level of this interface will be assessed under changing circumstances. It might not be required in the case of Java Launcher.
  • Debuggability of this code will be improved.
  • Removal of Launcher from the stop-domain/stop-instance logic: Unfortunately, I don't know why this is there to begin with. It makes no sense to go through Launcher to send an RMI-SHUTDOWN event to a running application server JVM.
             ______________         ______________       _______________
            | asadmin JVM  | ----- | Launcher JVM |-----| RMI ClientJVM |
            |______(1)_____|       |_____(2)______|     |_______(3)_____|
                   |                                         |
                   |                                         | RMI Shutdown Event
                   |                                         |  (current)
                   |                                     ____|__________
                   +----(proposed)RMI Shutdown Event--- | Running app   |
                                                        | server VM     |
                                                        |_______________|
                    

Whenever the server starts up an RMI Stub (A live object) is written to the server's config folder. If a piece of software can access this Stub, it can be an RMI client to the running server. So, while stopping the domain or server instance, all that's needed is the access to this stub. In fact, the logic through the Launcher does the same thing -- a Class called PEMain is called with an argument, stop and that becomes the RMI Client, to send a SHUTDOWN event to the running server. To achieve this, we don't need 3 JVM's. We just need one JVM and that is of the asadmin, while stopping the domain and that of node-agent, while stopping the server instance. This has huge benefits while removing the complexity. See Issue No. 949 for some hideous side effects of the way we do it now.

Synchronization

Background

This is the piece of code that is supposed to take care of bringing the cache repositories in sync with the central repository. For all practical purposes, the central repository = "config" subfolder of the domain's folder.

Changes planned for this release

Nazrul Islam is supposed to provide the details here.

 

Node Agent

Background

Node Agent controls the life cycle of the server instances on a given node. It optionally starts the instances, stops them, initiates synchronization of their repositories and attempts to restart the failed instances among other things.

Changes planned for this release

  • Handling of admin password change across the domain: Currently, when the admin password is changed, it is dynamically applicable only to the DAS The communication between node-agents/server-instances and DAS still continues to assume the older password through a cache of JMX connections. The upshot of this is that unless the DAS, all the node-agents and server instances are restarted after changing the admin password (providing the new admin password in the process), the communication between DAS and node agents/server instances will remain unpredictable. But having to restart all the entities in the domain after an admin-password-change operation seems illogical. It might be hard to take care of all the edge-cases for this release, when admin password is changed, but user experience needs to be improved in this case. Following is what will be attempted for this release in this regard:
    • Admin Interfaces (admin CLI, GUI, AMX) should clearly indicate to the administrator what needs to restarted and how, after the admin password is changed. It is important to stress that until the restart is done in the described fashion, the DAS and node-agent/server instances will continue to use older admin password for internal communication. Issue: I don't know what happens when the auth-realm for administrators is LDAP-realm. It is for this reason that we take a safe approach for this release.
  • Synchronize an instance on restarting the node-agent: We have found some cases where the administrators would like to forcefully sync the instance's cache repository with central repository on restarting the node-agent. Today, when a node-agent is restarted using asadmin stop-node-agent followed by asadmin start-node-agent, the synchronization does not happen. That's by design and cannot be overridden. But in some cases, an administrator would knowingly want to synchronize the instances, an option could be provided. For this release, we will provide an explicit option to synchronize the instances when the node-agent is restarted. Administrators must know however that when used, all the instances will be synchronized on node-agent's startup.
  • Improved restore of DAS from a backup(6380268): When a DAS is restored on some machine (after a machine failure), all the node-agents must be manually made aware of this change. For this release, we will make changes such that since the DAS knows the locations of all the node-agents (that have shaken hands with DAS) in the given domain, DAS will convey its own changed location to the node-agents. This way, node-agents don't have to be manually modified.
    • For this release, this applies to all the node-agents that are running at the time of restored DAS's startup. Any node-agents that are not running at that time will have to be manually modified to know DAS's changed location.
    • There is no explicit command/interface defined for this operation. [There might be implications of doing this implicitly, TBD ]. DAS performs this as a low-priority background task at the time of its own startup.
    • A node-agent establishes the trust with the new incarnation of the DAS if and only if:
      • DAS has the right admin user and admin password.
      • DAS sends the right server certificate that is already trusted by the node-agent. Note that a node-agent is either unbound or is bound to exactly one DAS A bound node-agent can be contacted by a DAS (or its reincarnation) only after there is at least one handshake between the two. Also note that every DAS has its own unique server certificate, by default.
  • Separation of client trust-store and server trust-store: Currently, we have a trust-store named .asadmintruststore in the user's home directory that is used to store the server certificates of all the secure domains that are contacted by asadmin over HTTPS, for administrative operations. As of now, the same trust-store is used by the node-agents. The plan is to use the server side trust store for server side communication between node-agents, server-instances and the DAS Using the domain-specific trust-stores for an intra-domain communication has several benefits including avoiding the accidental trust of a node-agent on a DAS that looks similar to the DAS that it has trusted at the time of binding (See 6450817).
Miscellaneous

Following miscellaneous changes will be done for this release.

  • Provision of sample resource creation templates for frequently used database vendors: This is to improve the usability of the asadmin add-resources command, which accepts an XML that defines the resources (See RFE - 582). Following templates will be provided. Appropriate database vendor specific properties will be provided, so that the users can use them as ready reference. For this release, only templates would be provided for following database vendors with appropriate resource types (e.g. JDBC Resource, JDBC Connection Pools defined in the template):
    • Oracle
    • JavaDB
    • MySQL
  • Attempt to do better integration with the operating platforms: Today, we have a command called create-service in asadmin that creates the so-called Operating System Services. We only have Solaris-10 SMF integration as of now. We will need to do following enhancements for this release. Creation of services pertains to Domain and node-agents. Once node-agents are integrated with services infrastructure on a particular platform, we automatically get the benefit for server instances.
    • Fix the issues 695, 726 blocking SMF integration on Solaris-10.
    • Provide etc/rc scripts for RedHat Linux. Integrate them with create-service command.
    • Revive and enhance the support for Windows. Integrate it with create-service command. We have an executable called appservService.exe, that is used for this purpose, but this approach is not usable.

 

4.2. Bug/RFE Number(s)

These are noted against the details above.

4.3. In Scope

The items covered in 4.1 are in scope.

4.4. Out of Scope

The items not covered in 4.1 are not in scope. Nothing in particular. Not all bugs and RFE's that will be fixed for this release can be mentioned here.

4.5. Interfaces

http://www.opensolaris.org/os/community/arc/policies/interface-taxonomy/ describes the permitted interface taxonomy.

4.5.1 Exported Interfaces

Interface Stability Former Stability (if changing) Comments
sun-domain_1_3.dtd (An XML file) EVOLVING EVOLVING The configuration schema governing the entire domain.
sun-resources_1_3.dtd (An XML file) EVOLVING EVOLVING The schema derived from sun-domain_1_3.dtd, and pertaining only to the resources. All the resource definitions in any XML used by asadmin add-resources must comply with this DTD.
New asadmin commands EVOLVING EVOLVING These are covered in the CLI one pager.
New AMX interfaces EVOLVING EVOLVING These will be covered in the AMX Javadocs.
processlauncher.xml UNSTABLE UNSTABLE Customers edit this file for some reasons and because we did not hide this file, it has become a sort of interface.



4.5.2 Imported interfaces

Interface Stability Exporting Project: Name, Specification or other Link. Comments
schema2beans.jar EVOLVING

NetBeans

Contains some config bean runtime classes (super classes). Pertains to NetBeans 5.5.
schema2beansdev.jar EVOLVING NetBeans Used for generation of config beans. Pertains to NetBeans 5.5.
jdmkrt.jar EVOLVING Java SE Java DMK for cascading feature.
Jakarta Commons Modeler EVOLVING Apache Used for Config MBeans/Model MBean generation support.

4.5.3 Other interfaces (Optional)

Not applicable.

 

4.6. Doc Impact

Moderate impact.

4.7. Admin/Config Impact

Administration specification, this is.

4.8. HA Impact

Not applicable.

4.9. I18N/L10N Impact

Not applicable.

4.10. Packaging & Delivery

Standard packages, zip files that are documented in the packaging specification.

4.11. Security Impact

Not applicable.

4.12. Compatibility Impact

TBD.

4.13. Dependencies

5. Reference Documents

// List of related documents, if any (BugID's, RFP's, papers, Blogs).
// Explain how/where to obtain the documents, and what each
// contains, not just their titles.

6. Schedule

6.1. Projected Availability

Covered elsewhere.

7. Document History

Version Date Author, Comment
1.0 06 January 2007 Incorporated some feedback. Made changes for features that could not make it.
0.91 01 September 2006 Kedar Mhaswade, Added material related to platform services.
0.9 31 August 2006 Kedar Mhaswade. Added material, Made it ready for review.
0.8 29 August 2006 Kedar Mhaswade, created.