\section{Article 1: IBM study finds more than half of midmarket companies lack an integrated digtal strategy}
This article discusses about the lack of strong digital strategy in small or midsize companies. The digital strategy focuses on fusing of physical and digital resources in a company. This is mainly the job of CMO or CIO (information). In the age of Cloud computing, social networking and mobile computing, digital strategy becomes more and more important to compnay's strategic advantage over rivals.
The article gives two examples, in a social music industry and in a digital medical industry. Social music company, music mastermind, aims at enabling average person's ability to produce their own music. This marks the recent shift in music producing from traditional content provider, manufacture to average social users. To a tech startup, this promises new opportunities, as proven by the popularity of Web 2.0. Yet it also poses challenges. To a small company, to serve large scale userbase, it is not an easy task. And this is where IBM's Cloud solution comes in.
The same thing applies to digital medical industry, which now undergoes a revolution that digitalizes every physical medical records and personal DNA information. This promises better personal treatment, but also raises controversy like privacy issues when dealing doctor-patient exchange. The goal of the company, Coriel Life Science, is to make use of personal genomic data to better tailor the medical care. Security and privacy involved in this process, when the company wants to outsource the patient's personal and private genimic data (which is also huge) to a third party Cloud computing company, like IBM.
\section{Article 2: Salesforce denies any post-PRISM cloud trust problems}
Basically this article is a clarification from Salesforce to confirm the security of their Cloud platform. Security in third party Cloud causes a lot controversy especially after the revellation of PRISM incident, a US massive surveillance programme. Saleforces is a Cloud computing company which helps midsize company to utilize their sales data; It stores different kinds of company data, most being sensitive, including customer complaints, sales forecase and product information.
\section{Article 3: Presto: Interacting with petabytes of data analytics in Facebook}
This articles introduces presto: a SQL query engine that emphasizes the interactive query processing on big data.
Nowadays, typical workload that is run in Cloud includes the batch processing and real-time data serving. Their application ranges from simply data storing/retrieving to complex analytics with machine learning and data mining. In this domain, Presto is a computation engine that is optimized towards real-time interactive queries.
Architecture-wise, Presto is not MapReduce, but instead a stream processing engine to me. It looks like a distributed in-memory stream processing system, in which each stage models are with operators designed to support SQL semantics.
PL/System-wise, Presto is written in Java. The good side is Java is fast for development. However flip side is that Java add a JVM layer on top of the native operating system on where it is running. In order for high performance java system programming, the engineers in Facebook comes up with tricks to avoid low level inefficiency related to memory allocation and garbage collection.
Extensibility: Presto is computation engine which can work together with other Cloud systems. Most notably with HBase. Presto can be imposed on top of HBase to collectively support a fault-tolerant/persistent database with interactive performance for complex SQL queries. One task in their roadmap is on a high-performance HBase connector.
Presto is open sourced.
\section{Article 4: HBase Performance Testing}
This article addresses the performance evaluation for HBase. The writing is not very formal, yet the takeaway I found useful in this article is that
1.it confirms that HBase is write optimized system.
2.the methodology to do system evaluation includes multiple designs; typical ones include using benchmark to simulate normal or peak production conditions, stress testing which tests with a harder condition and may saturate the system, and sustained test which usually run test in long enough time in order to see behaves that does not surface in short time (e.g. does java GC come into play, does compaction occur?)
3.in HBase, most workload is not CPU bounded, but bounded by factor like network, diskIO, memoryLocking...
\section{Compare and Contrast}
These articles talk about emerging shift towards Cloud computing, including fusing digital and physical resource/data in small size company, security issues in Cloud in the post-PRISM age, a computation engine released by Facebook for realtime analysis in Cloud, and performance evaluation of Cloudy storage software HBase.
11/26/2013
12/01/2009
jimmy lin's talk
observation:
the more data, the better performance u get.
problem (why is this different):
impl. issues: concurrency.
should have a simple programming model, and large scale resources available to the masses.
PS:
break down into two problems
On y axis, you have XXX
6/11/2009
百度面试 C/C++
-C和C++的区别
-操作符？类型转换的种类
:实参到形参的类型转换有四种，有一些优先顺序。有四种exact match, promotion, standard conversion, class-type conversion, 见7.8.4
-文件声明中的__cdecl, __pascal, __stdcall是干什么的
:简单而言，这些是关键字。在函数声明的时候用在函数名前而返回值之后，如extern Region __cdecl XCreateRegion。
:__cdecl用于标准的C函数调用，__stdcall是C++调用，__pascal则是WINAPI的函数调用。具体意义见网上参考，如下：
http://hi.baidu.com/tendollor/blog/item/1584f20facefe6226059f383.html
http://hi.baidu.com/aniu_home/blog/item/0e595ad8ba8a513432fa1c6d.html
-template的偏特化
:特化specialization：使用template的函数对某些类型有特殊的实现，使用特化。template<>
?重载和特化的区别：重载的调用对实参有类型转换，特化的函数调用没有
:偏特化是对部分形参进行特化
-操作符？类型转换的种类
:实参到形参的类型转换有四种，有一些优先顺序。有四种exact match, promotion, standard conversion, class-type conversion, 见7.8.4
-文件声明中的__cdecl, __pascal, __stdcall是干什么的
:简单而言，这些是关键字。在函数声明的时候用在函数名前而返回值之后，如extern Region __cdecl XCreateRegion。
:__cdecl用于标准的C函数调用，__stdcall是C++调用，__pascal则是WINAPI的函数调用。具体意义见网上参考，如下：
http://hi.baidu.com/tendollor/blog/item/1584f20facefe6226059f383.html
http://hi.baidu.com/aniu_home/blog/item/0e595ad8ba8a513432fa1c6d.html
-template的偏特化
:特化specialization：使用template的函数对某些类型有特殊的实现，使用特化。template<>
?重载和特化的区别：重载的调用对实参有类型转换，特化的函数调用没有
:偏特化是对部分形参进行特化
4/14/2009
Install VS 2008 on a Windows server (Sever 2003)
VS 2008 is to be installed on a 64-bit Windows Server 2003. Yet the VS Web Authoring Component continues to report problems (of HRESULT "-2147023293")...
As in [2], no use to look at the log in "dd_error_vs_vstscore_90.txt", which is quite high level. Yet what is useful is in %temp%SetupExe(***).log. The log says "Error: failed MsiEnumProducts ErrorCode: 2(0x2)."
A solution [1] can be located by googling the error, which perfectly solves the problem...
references:
1.http://support.microsoft.com/kb/954361
2.http://blog.mediawhole.com/2008/07/installing-visual-studio-2008.html
As in [2], no use to look at the log in "dd_error_vs_vstscore_90.txt", which is quite high level. Yet what is useful is in %temp%SetupExe(***).log. The log says "Error: failed MsiEnumProducts ErrorCode: 2(0x2)."
A solution [1] can be located by googling the error, which perfectly solves the problem...
references:
1.http://support.microsoft.com/kb/954361
2.http://blog.mediawhole.com/2008/07/installing-visual-studio-2008.html
4/06/2009
多线程编程-共享变量的一致性
线程存储模型和变量映射
本质上，每个线程有自己完全独立的寄存器，但每个线程被允许共享进程的所有存储器空间。具体而言，读写区域（存储global variable, local static variable）和堆都是直接可以在thread routine里访问的（堆而言，只要指针能传进去）。栈而言(存储local variable)，一般线程独立访问，但每个线程栈之间不设防：如也可以通过全局变量的方式让其他线程访问local variable.
共享变量的一致性：
当有多个线程访问一个变量实例（一个变量可能在不同的上下文（如方程，线程）有多个实例），可能导致不一致的现象。（系统对不同thread routine的汇编代码做interleaving的，且寄存器是完全独立的）
问题解决：通过信号量/互斥锁来同步线程，使得在临界区内只有一个线程。关键是对信号量的更新操作是原子的，不可中断的（应该是系统级别实现的）。
对于共享资源（如thread pool），如生产者/消费者模型，其本质和共享变量是相同的（相当于两个write操作）。
PS:
小细节：线程detach和joinable
本质上，每个线程有自己完全独立的寄存器，但每个线程被允许共享进程的所有存储器空间。具体而言，读写区域（存储global variable, local static variable）和堆都是直接可以在thread routine里访问的（堆而言，只要指针能传进去）。栈而言(存储local variable)，一般线程独立访问，但每个线程栈之间不设防：如也可以通过全局变量的方式让其他线程访问local variable.
共享变量的一致性：
当有多个线程访问一个变量实例（一个变量可能在不同的上下文（如方程，线程）有多个实例），可能导致不一致的现象。（系统对不同thread routine的汇编代码做interleaving的，且寄存器是完全独立的）
问题解决：通过信号量/互斥锁来同步线程，使得在临界区内只有一个线程。关键是对信号量的更新操作是原子的，不可中断的（应该是系统级别实现的）。
对于共享资源（如thread pool），如生产者/消费者模型，其本质和共享变量是相同的（相当于两个write操作）。
PS:
小细节：线程detach和joinable
1/20/2009
Pip:detecting the unexpected in distributed systems.
title: Pip:detecting the unexpected in distributed systems
venue: nsdi 2006
authors: ucsd
Given a running distributed system, how to determine whether or not there is unexpected behavior so far and if yes, where it is? For pip, the basic idea is to capture necessary system behavior online and to run a offline algorithm to check them against user's expectation. (Of coz, there are some options in making such choice, like why not check the unexpected online? why not check all-possible behaviors?)
In such a framework, several detailed issues are posed:
- how to describe behavior of of a distributed system?
- how to express user's expectation for a distributed system?
- how to capture system behavior? what need to capture and what not? (This is left to users and automatic analysis tools (say, mace), by an annotation lib to indicate the PathId for an event.)
1. Behavior model - How to describe behavior of a distributed system?
System behavior is described by a series of path instances, each of which is raised by an outside input (say, a request). Each path instance portraits the internal procedure in the system to answer or to finish the task. It consists of a series of events, organized by their natural causality, (thereby forming a DAG). In a distributed setting, an event could be a task which resides on a single thread (thus on a single host), or a message which help communicate between threads or hosts.
2. Expressing expectation and checking algo.
Given a bunch of path instances and a set of user expectation, Pip checks whether each path could satisfy any of user expectations.
2.1)
Each expectation is a path-validator/invalidator. Each validator consists of several thread patterns, each indicating how a desired thread should run. Within a thread pattern, there are statements and some flexible descriptions of statement flow. A statement is the basic unit of user expectation, including tasks, messages, and notices. Those descriptions include xor and future primitives, flexible enough to capture various program/runtime flows.
The whole technical point is how to model distributed behavior and how to describe it by an expressive language. The difficulty is how to express the prevalent parallelism in every aspect of distributed systems. In essence, how to model a DAG in a language. In this respect, the expectation language is in natural parallel--- no particular order is imposed between thread patterns; within a thread, there are flexible primitives allowing expression of various runtime flow.
2.2) the checking algo.
Given a path instance and an expectation (expressed by a code snippet), checking algo. is to determine whether the path instance follows the expectation. The point here is that the expectation language is design to be expressive and flexible enough, making the matching difficulty. Specifically, the ...
venue: nsdi 2006
authors: ucsd
Given a running distributed system, how to determine whether or not there is unexpected behavior so far and if yes, where it is? For pip, the basic idea is to capture necessary system behavior online and to run a offline algorithm to check them against user's expectation. (Of coz, there are some options in making such choice, like why not check the unexpected online? why not check all-possible behaviors?)
In such a framework, several detailed issues are posed:
- how to describe behavior of of a distributed system?
- how to express user's expectation for a distributed system?
- how to capture system behavior? what need to capture and what not? (This is left to users and automatic analysis tools (say, mace), by an annotation lib to indicate the PathId for an event.)
1. Behavior model - How to describe behavior of a distributed system?
System behavior is described by a series of path instances, each of which is raised by an outside input (say, a request). Each path instance portraits the internal procedure in the system to answer or to finish the task. It consists of a series of events, organized by their natural causality, (thereby forming a DAG). In a distributed setting, an event could be a task which resides on a single thread (thus on a single host), or a message which help communicate between threads or hosts.
2. Expressing expectation and checking algo.
Given a bunch of path instances and a set of user expectation, Pip checks whether each path could satisfy any of user expectations.
2.1)
Each expectation is a path-validator/invalidator. Each validator consists of several thread patterns, each indicating how a desired thread should run. Within a thread pattern, there are statements and some flexible descriptions of statement flow. A statement is the basic unit of user expectation, including tasks, messages, and notices. Those descriptions include xor and future primitives, flexible enough to capture various program/runtime flows.
The whole technical point is how to model distributed behavior and how to describe it by an expressive language. The difficulty is how to express the prevalent parallelism in every aspect of distributed systems. In essence, how to model a DAG in a language. In this respect, the expectation language is in natural parallel--- no particular order is imposed between thread patterns; within a thread, there are flexible primitives allowing expression of various runtime flow.
2.2) the checking algo.
Given a path instance and an expectation (expressed by a code snippet), checking algo. is to determine whether the path instance follows the expectation. The point here is that the expectation language is design to be expressive and flexible enough, making the matching difficulty. Specifically, the ...
1/09/2009
Networked DHTs - Network Background
Today, various networks/protocols are in existence. Different classification methods/viewpoints are essential in understanding them.
Routing principles(for routing protocol only):There are generally three kinds of routing protocols: the link-state and the distance-vector (including path-vector).
LS: advertise each link info. to all other nodes. (flooding, msg per router: O(n*d))
-essentially, many small messages
DV: advertise its whole routing table to only neighbors. (rummer, msg per router: O(d))
-essentially, relatively few large messages
Note the number of advertisement is meant to be before the computation.
While link-state involves global flooding of each link to update "indexed" route info., distance-vector is to disseminate local distance-vector through neighbors. The computation of LS is after flooding, whereas DV embeds the computation within the dissemination process (ie., in a round-to-round manner). Specifically, the link-state is to flood every link state to every node in the network, in a way to construct a global view of network topology known by every node. The flooding is always loop-free by indicating a sequence number. The routing table is then constructured by computing the Shortest-Paths locally by the Dijkstra. The distance-vector is a every-node-run-it and round-to-round version of the Bellman-ford algorithm; Every node maintains a distance vector to every other node in the network (note, not the global network topology), and update it by disseminate it through all neighbors to keep other's DV up-to-date. ^2
Exterior Gateway Protocol is BGP which is kind of DV (actually path-vector, and at application layer). Examples of Interior Gateway Protocol (for LAN) may be RIP, a distance-vector protocol (application/network layer), or OSPF, a typical link-state protocol (link layer).
Protocol Layering Overview:
Each kind of network has/is a suit of protocols, with each belonging to a/some specific layer in the protocol stack. For example, the Ethernet widely used in enterprise internal networks mainly focus on layer-1 and layer-2, while the Internet protocols are scattered in 4 layers (which can be essentially mapped to all 7 layers of the OSI model). The Internet Protocols has a waist model?...
PS: Internet could also refer to the set of all existing networks.
LS: advertise each link info. to all other nodes. (flooding, msg per router: O(n*d))
-essentially, many small messages
DV: advertise its whole routing table to only neighbors. (rummer, msg per router: O(d))
-essentially, relatively few large messages
Note the number of advertisement is meant to be before the computation.
While link-state involves global flooding of each link to update "indexed" route info., distance-vector is to disseminate local distance-vector through neighbors. The computation of LS is after flooding, whereas DV embeds the computation within the dissemination process (ie., in a round-to-round manner). Specifically, the link-state is to flood every link state to every node in the network, in a way to construct a global view of network topology known by every node. The flooding is always loop-free by indicating a sequence number. The routing table is then constructured by computing the Shortest-Paths locally by the Dijkstra. The distance-vector is a every-node-run-it and round-to-round version of the Bellman-ford algorithm; Every node maintains a distance vector to every other node in the network (note, not the global network topology), and update it by disseminate it through all neighbors to keep other's DV up-to-date. ^2
Exterior Gateway Protocol is BGP which is kind of DV (actually path-vector, and at application layer). Examples of Interior Gateway Protocol (for LAN) may be RIP, a distance-vector protocol (application/network layer), or OSPF, a typical link-state protocol (link layer).
Protocol Layering Overview:
Each kind of network has/is a suit of protocols, with each belonging to a/some specific layer in the protocol stack. For example, the Ethernet widely used in enterprise internal networks mainly focus on layer-1 and layer-2, while the Internet protocols are scattered in 4 layers (which can be essentially mapped to all 7 layers of the OSI model). The Internet Protocols has a waist model?...
PS: Internet could also refer to the set of all existing networks.
Network topological/protocol hierarchy:
host -> bridges and segments (Ethernet) -> routers and subnets -> ASes (hierarchical)
Network topology is truly hierarchical in nature: At the toppest level (application layer), AS is hierarchical; Each lowest-Tier AS could then contain several subnets and routers (a subnet could be an Ethernet); Again, each modern Ethernet could consist of several Ethernet segments connected by switches or bridges. An Ethernet segment is shared-bus structure of several hosts.
Above network layer, the Internet is really a set of subnets and routers between subnets. Subnets are identified by a single network ip, which means network ip appears in routing table's entries. By such means, router store per-subnet routing info, rather than per-host. At network layer, the internal structure of LAN is simly full-connected; every host can contact its attach router in one hop.
As to the host IP address within one subnet, there are two forms; the host is connected to the router, directly, or by NAT. The former implies each host is directly on the Internet (在公网上), having a public IP address. In the latter case, the LAN's network identifier is a complete IP, yet each host is identified on the Internet by its public identifier, namely LAN's IP+port, dynamically. From internal view, each host is configured with a reserved IP, which is transformed to its public identifier by NAT. This solution is to resolve the scarcity of public IP on the Internet.
On application layer, certain set of subnets and routers could form an Autonomous System (AS). The hierarchy of ASes could be recursive; set of some ASes could form a high-level AS. Each ISP corresponds to ISP who administrates this AS and has right in designating routing pollicy within this AS. The hierarchy of ASes can be modeled as a DAG (not the tree). The provider-customer relationship forms a tree, while multihoming, peering could be other links in the DAG besides the tree. Routing at application layer could be divided as inter-domain and intra-domain routings.
Link layer takes a closer look than network and explains how a single hop in network is performed. Specifically, link layer focuses on the internal structure of LAN and the link between routers (ie., point-to-point link). The core technical issue of link layer (including data-link and physical layers) is how to route (covered by data-link layer) following the topology of physical layer. The data-link layer focuses on Media Access Control and Logic Link Control, the form requiring to identify a host (whether it is the host of destination) and the latter providing some routing quality guarantee. Thus, a major distinct of data-link layer is to route on MAC, which introduces problems involving translation of IP to MAC (ARP) and automatically assigning IP (DHCP).
Ethernet is a typical LAN with protocols on link layer. The network topology is bridging (or switching) several segments each of which is a shared bus accessed by CSMA/CD, (note all these are at physical layer). The routing involves flooding and source-learning. The flooding (partially in unicast and always in broadcast) could form a loop, which Ethernet resolves by computing the spanning tree.
How each frame corresponds to a data packet?
Why IGMP should be a layer-3 protocol, rather than a layer-2 one, since it actually works between router and end-hosts?(because it doesn't run on switches!)
host -> bridges and segments (Ethernet) -> routers and subnets -> ASes (hierarchical)
Network topology is truly hierarchical in nature: At the toppest level (application layer), AS is hierarchical; Each lowest-Tier AS could then contain several subnets and routers (a subnet could be an Ethernet); Again, each modern Ethernet could consist of several Ethernet segments connected by switches or bridges. An Ethernet segment is shared-bus structure of several hosts.
Above network layer, the Internet is really a set of subnets and routers between subnets. Subnets are identified by a single network ip, which means network ip appears in routing table's entries. By such means, router store per-subnet routing info, rather than per-host. At network layer, the internal structure of LAN is simly full-connected; every host can contact its attach router in one hop.
As to the host IP address within one subnet, there are two forms; the host is connected to the router, directly, or by NAT. The former implies each host is directly on the Internet (在公网上), having a public IP address. In the latter case, the LAN's network identifier is a complete IP, yet each host is identified on the Internet by its public identifier, namely LAN's IP+port, dynamically. From internal view, each host is configured with a reserved IP, which is transformed to its public identifier by NAT. This solution is to resolve the scarcity of public IP on the Internet.
On application layer, certain set of subnets and routers could form an Autonomous System (AS). The hierarchy of ASes could be recursive; set of some ASes could form a high-level AS. Each ISP corresponds to ISP who administrates this AS and has right in designating routing pollicy within this AS. The hierarchy of ASes can be modeled as a DAG (not the tree). The provider-customer relationship forms a tree, while multihoming, peering could be other links in the DAG besides the tree. Routing at application layer could be divided as inter-domain and intra-domain routings.
Link layer takes a closer look than network and explains how a single hop in network is performed. Specifically, link layer focuses on the internal structure of LAN and the link between routers (ie., point-to-point link). The core technical issue of link layer (including data-link and physical layers) is how to route (covered by data-link layer) following the topology of physical layer. The data-link layer focuses on Media Access Control and Logic Link Control, the form requiring to identify a host (whether it is the host of destination) and the latter providing some routing quality guarantee. Thus, a major distinct of data-link layer is to route on MAC, which introduces problems involving translation of IP to MAC (ARP) and automatically assigning IP (DHCP).
Ethernet is a typical LAN with protocols on link layer. The network topology is bridging (or switching) several segments each of which is a shared bus accessed by CSMA/CD, (note all these are at physical layer). The routing involves flooding and source-learning. The flooding (partially in unicast and always in broadcast) could form a loop, which Ethernet resolves by computing the spanning tree.
How each frame corresponds to a data packet?
Why IGMP should be a layer-3 protocol, rather than a layer-2 one, since it actually works between router and end-hosts?(because it doesn't run on switches!)
References:
1. Ion's lecture: http://www-inst.eecs.berkeley.edu/~ee122/fa08/
3. LS vs DV: http://books.google.com/books?...
订阅：
博文 (Atom)