Oracle Enterprise Manager Java Application Diagnostic Expert (JADE) improves the availability, performance and provides in-depth diagnostics for Java applications with low overhead. It enables:

1. Production diagnostics with no application instrumentation, saving time in reproducing problems
2. Visibility into Java activity including in-flight transactions, allowing administrators to proactively identify issues rather than diagnosing after-the-fact (application hangs, crashes, memory leaks, locks)
3. Tracing of transactions from Java to Database and vice-versa, enabling faster resolution of problems that span different tiers
4. Differential heap analysis in production applications.

This is Auptyma‘s “Java Application Monitor”. It’s likely Auptyma was acquired by Oracle – its founder and CEO was previously Director of the CRM Performance Group at Oracle.

…similar solutions for addressing the scalability challenges:

On the Data Tier we see the following:

1. Adding a caching layer to take advantage of memory resources availability and reduce I/O overhead

2. Moving from a database-centric approach to partitioning, aka shards

On the Business Logic Tier:

3. Adding parallelization semantics to the application tier (e.g., MapReduce)

4. Moving to scale-out application models to achieve linear scalability

5. Moving away from the classic two-phase commit and XA for transaction processing (See: Lessons from Pat Helland: Life Beyond Distributed Transactions)

… a repository of information collected from one run and used on another run is on the list for things we want to do in Dolphin. It’s actually on the runtime groups list but we will certainly take advantage of it. There has also been talk of using annotations to give the jit a hint. This actually isn’t too popular since it is too much like “register” declarations in C. It’s only a hint and is too often abused.

1. High Performance Throughput Computing, Dr. Marc Tremblay, Sun Fellow, Vice President, and Chief Architect, Sun’s Scalable Systems Group
2. Software and the Concurrency Revolution, Herb Sutter, Microsoft
3. Architecture Support for Parallel Programming, Dr. Kunle Olukotun, Associate Professor, Stanford University

Uniform Utilization of hardware resources ensures scalability:

• Uniform Hardware Utilization Through Time (All program Phases use the hardware uniformly)
• Uniform Utilization of Hardware Threads (All Hardware Threads are Busy)
• Uniform Use of Software Threads (There are enough threads)
• Uniform Cache Set Use in the Cache Hierarchy (No cache sets are over subscribed)
• Uniform Physical Memory Use (All memory boards used evenly)
• Uniform Virtual Memory Use (No Hot Locks)

(Q): If you have to pick top 10 things that you must monitor on any server to look for performance and/or scalability issues…what would they be?
Richard McDougall (A): Off the top of my head, in no particular order:

1. CPU: Check idle time and run queue length.
2. If there’s a CPU bottleneck, check if it’s an application or kernel CPU utilization issue with mpstat: high percentages of users indicate it’s an application issue. High sys may point to high network load or lock contention.
3. Memory: Check MDBs memstat to ensure there is sufficient free memory
4. Network: Check that networks are not overloaded by observing the bytes xfered against the availability bandwidth per link.
5. CPU for network: check if any CPUs are 100% busy servicing network interrupts. CPUs at 100% in mpstat, or intrstat are possible candidates.
6. File system latency: check the application visible latency with DTrace at the system call level (perhaps fsstat, iosnoop, or an aggregation around system calls).
7. Storage latency: check disk latency with iostat
8. Application level lock contention: check application level locks are now visible with plockstat
9. Kernel level locks: Check for hot locks with lockstat.
10. Check MMU activity on SPARC using trapstat. Sometimes an application may be reporting as running 100% in user mode, but may actually be spending a significant amount of time in kernel mode servicing TLB misses. Trapstat will show the % of time spent using TLB misses. If a significant amount of time (>10%) is evident, then large MMU pages may help.

… user authentication is a transactional event that requires fast, reliable, scalable, persistence and robust high-availability. To deliver this level of service, Google Accounts uses Berkeley DB HA for the storage and retrieval of user data and for replication, thereby ensuring scalability and availability.