Hadoop is a powerful platform for data analysis and processing, but many struggle to understand how it fits in with regard to existing infrastructure and systems. A series of common integration points, technologies, and patterns are defined and illustrated in this presentation. [slides]

Cary Milsap, “Optimizing Oracle Performance” (O’Reilly, 2003). On p. 361, there is a definition of scalability expressed in terms of the response time (R). [ed. scalability: The rate of change of response time with respect to some specified parameter. For example, one may speak of the scalability of a query with respect to the number of rows returned, the scalability of a system with respect to the number of CPUs installed, and so on.] This stands in contrast to the way I developed my Universal Scalability Law, which is based on a kind of normalized throughput. … For those of you who haven’t seen it, here’s how it works …

…similar solutions for addressing the scalability challenges:

On the Data Tier we see the following:

1. Adding a caching layer to take advantage of memory resources availability and reduce I/O overhead

2. Moving from a database-centric approach to partitioning, aka shards

On the Business Logic Tier:

3. Adding parallelization semantics to the application tier (e.g., MapReduce)

4. Moving to scale-out application models to achieve linear scalability

5. Moving away from the classic two-phase commit and XA for transaction processing (See: Lessons from Pat Helland: Life Beyond Distributed Transactions)

… steps in optimizing and tuning mysql is to identify the queries that are causing problems. How can we find out what queries are taking a long time tocomplete? How can we see what queries are slowing down the mysql server? Mysql has the answer for us and we only need to know where to look for it…

Most scaling and performance challenges are almost always related to the data layer, and are common across all language platforms. Even as a self-proclaimed PHP evangelist, I was very startled to find out that all of the theories I was subscribing to were true. There is simply no truth to the idea that Java is better than scripting languages at writing scalable web applications. I won’t go as far as to say that PHP is better than Java, because it is never that simple. However it just isn’t true to say that PHP doesn’t scale, and with the rise of Web 2.0, sites like Digg, Flickr, and even Jobby are proving that large scale applications can be rapidly built and maintained on-the-cheap, by one or two developers.

Let’s not confuse the Java runtime with the language itself, I’m sure one can write “scalable” apps in Logo as well as long as the deployment is horizontal.
The truth is the comparisons are bogus, Java and scripting languages are more complimentary than competitive. Now if/when Java runtimes will add support for dynamically typed languages (the ‘Invokeddynamic’ bytecode), we’ll probably see a noticeable rise in adoption of hosted (JRuby, Jython) scripting languages within the “enterprise”, as RAS and deployment issues will no longer be the (biggest?) stumbling blocks they are now.

• Steve Pawlowski and Ofri Wechsler: Intel® Coreâ„¢ microarchitecture and Usages
• Paolo A. Gargini: Intel’s Silicon R&D Pipeline
• The keynotes of Pat P. Gelsinger (Digital Enterprise), Sean M. Maloney (Mobility), Donald J. MacDonald (The Digital Home) and Justin R. Rattner (Opening Keynote)

1. High Performance Throughput Computing, Dr. Marc Tremblay, Sun Fellow, Vice President, and Chief Architect, Sun’s Scalable Systems Group
2. Software and the Concurrency Revolution, Herb Sutter, Microsoft
3. Architecture Support for Parallel Programming, Dr. Kunle Olukotun, Associate Professor, Stanford University

Uniform Utilization of hardware resources ensures scalability:

• Uniform Hardware Utilization Through Time (All program Phases use the hardware uniformly)
• Uniform Utilization of Hardware Threads (All Hardware Threads are Busy)
• Uniform Use of Software Threads (There are enough threads)
• Uniform Cache Set Use in the Cache Hierarchy (No cache sets are over subscribed)
• Uniform Physical Memory Use (All memory boards used evenly)
• Uniform Virtual Memory Use (No Hot Locks)

(Q): If you have to pick top 10 things that you must monitor on any server to look for performance and/or scalability issues…what would they be?
Richard McDougall (A): Off the top of my head, in no particular order:

1. CPU: Check idle time and run queue length.
2. If there’s a CPU bottleneck, check if it’s an application or kernel CPU utilization issue with mpstat: high percentages of users indicate it’s an application issue. High sys may point to high network load or lock contention.
3. Memory: Check MDBs memstat to ensure there is sufficient free memory
4. Network: Check that networks are not overloaded by observing the bytes xfered against the availability bandwidth per link.
5. CPU for network: check if any CPUs are 100% busy servicing network interrupts. CPUs at 100% in mpstat, or intrstat are possible candidates.
6. File system latency: check the application visible latency with DTrace at the system call level (perhaps fsstat, iosnoop, or an aggregation around system calls).
7. Storage latency: check disk latency with iostat
8. Application level lock contention: check application level locks are now visible with plockstat
9. Kernel level locks: Check for hot locks with lockstat.
10. Check MMU activity on SPARC using trapstat. Sometimes an application may be reporting as running 100% in user mode, but may actually be spending a significant amount of time in kernel mode servicing TLB misses. Trapstat will show the % of time spent using TLB misses. If a significant amount of time (>10%) is evident, then large MMU pages may help.

… user authentication is a transactional event that requires fast, reliable, scalable, persistence and robust high-availability. To deliver this level of service, Google Accounts uses Berkeley DB HA for the storage and retrieval of user data and for replication, thereby ensuring scalability and availability.